Phenquestions
Why mine Twitter?
Twitter is a perfect place to pull data from. Here are some reasons why this is true:
- All the tweets on Twitter are public
- The new data keeps coming in real-time, this means, you never run out of fresh data to test on
- Twitter has interesting API for developers to pull data from, they are friendly as well
- APIs provided by Twitter provide easy ways to pull tweets related to a specific user or to a specific word/hashtag or tweets in a certain location
Getting Started
Now that we have justified why we're using this platform, let's start collecting our tools. We'll be using Java 8 for this lesson but feel free to use above versions (though some tweaks, not tweets, might be needed to use them).
We will be using a Java library called Twitter4J to connect to Twitter's API.
Getting Twitter API key
Getting Twitter API key is necessary to access its data as this is how Twitter keeps track of the data and the request count our application makes to Twitter.
Let's create a Twitter app and get the correct keys to move forward.
- Create an app here
In above form, create an Application with a unique name, a website name (use a placeholder website if you don't have one), and a project description. Accept the terms and conditions (if you do) and proceed to the next page.
Once the project is created, you should see a page with following header:
- In information below, click on 'Keys and Access Tokens' tab to get the following information:
- Scroll down and click on “Generate Access Tokens” to get below information:
We will need these values later so it'll be better to keep this tab open.
Getting Started with Twitter4J
Twitter4J is an unofficial Java library for the Twitter API. With Twitter4J, we can easily integrate our Java application with the Twitter service.
Maven Dependency
To start, we'll add appropriate Maven Dependency to our Java project.
Find latest maven dependency version here.
Authentication
We have added required Maven dependency now. It's time that we start talking to the API and Twitter.
To start the conversation, we need to authenticate our calls to Twitter so that it knows that only a known user is accessing the data. For this, let's setup our keys we obtained earlier.
static final String CONSUMER_KEY = "you-key";static final String CONSUMER_SECRET = "secret";
static final String ACCESS_TOKEN = "token";
static final String ACCESS_TOKEN_SECRET = "token-secret";
public static Twitter getTwitterInstance()
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)bashbash
.setOAuthConsumerKey(CONSUMER_KEY)
.setOAuthConsumerSecret(CONSUMER_SECRET)
.setOAuthAccessToken(ACCESS_TOKEN)
.setOAuthAccessTokenSecret(ACCESS_TOKEN_SECRET);
TwitterFactory tf = new TwitterFactory(cb.build());
return tf.getInstance();
Example : Showing Timeline
In this example, we'll be showing some most recent tweets from authenticated user's timeline. We'll do this by using the Twitter's object Status instance as:
private static void showHomeTimeline(Twitter twitter)List
try
statuses = twitter.getHomeTimeline();
System.out.println("Showing home timeline.");
for (Status status : statuses)
System.out.println(status.getUser().getName() + ":" + status.getText());
String url= "https://twitter.com/" + status.getUser().getScreenName() + "/status/"
+ status.getId();
System.out.println("Above tweet URL : " + url);
catch (TwitterException e)
e.printStackTrace();
The result should look like a bunch of random tweets:
Following the link to the tweet will often bring you to the tweet itself. Following the link from the first tweet would give us the following result:
Apart from the username and the tweet text, the Twitter API has a lot of information to give which can be inferred from following available methods:
status.getSource();status.getCreatedAt();
status.getFavoriteCount();
status.getGeoLocation();
status.getLang();
status.getPlace();
status.getRetweetCount();
status.getUser().getBiggerProfileImageURL();
status.getUser().getEmail();
status.getUser().getFollowersCount();
status.getUser().getFriendsCount();
This gives a lot of information related to the Tweet and the user who posted the tweet. These include not all methods, feel free to explore all the methods available.
Note that these attributes can be extremely useful if your application depends on more data.
Example : Post a Tweet
In this example, we'll simply post a new tweet from our code as the user is already authenticated. Let's put some sample code here:
private static void updateTweet(Twitter twitter, String tweet) throws TwitterExceptionStatus status = twitter.updateStatus(tweet);
System.out.println("Successfully updated the status to [" + status.getText() + "].");
Posting a new tweet is as simple as that.
Example : Tweets from a Specific User
It is very easy to get another user tweets, just pass a username and the API will return some recent tweets for the user.
Let's try pulling the latest twenty tweets from twitter account @linuxhint:
Here is the sample code:
Listfor (Status status : statuses)
String fmt = "@" + status.getUser().getScreenName() + " - " + status.getText();
System.out.println(fmt);
When you run this program, you should see Tweets for LinuxHint.
Popular applications of this type of data can include:
- Running analysis on specific users, and how they interact with the world
- Finding Twitter influencers and analyzing their follower trends and interactions
- Monitoring the changes in the followers of a user
Example: Finding Tweets Using a Keyword
Let's do one last example: Getting the most recent tweets that contain a keyword. This can be extremely useful if you want to monitor specifically mentioned topics in the Twitter world, or even to see how your business is getting mentioned.
Let's say we want to see how Twitter's been mentioning Linux:
//searchTerm=Linuxprivate static void searchTweets(Twitter twitter, String searchTerm) throws TwitterException
Query query = new Query("source:" + searchTerm);
QueryResult result = twitter.search(query);
for (Status status : result.getTweets())
System.out.println("@" + status.getUser().getScreenName() + ":" + status.getText());
Here are some practical ways you can use this information:
- Create a spatial graph on where your company is mentioned the most around the world
- Run sentiment analysis on tweets to see if the overall opinion of your company is positive or negative
- Create a social graphs of the most popular users that tweet about your company or product
We can cover some of these topics in future articles.
Twitter's API is immensely useful in data mining applications, and can provide vast insights into the public opinion.
