Geolocation Twitter

 Oscar-Rydh Merge branch 'master' of https://github.com/JoelKlint/twitter-geoloca… 

	data	Created layer for creating bounding boxes based on tweet langs	2 months ago
	database	Changed so that the database runs layers on all data but only display…	2 months ago
	layers_method	Fixed annoying enviroment varaible	2 months ago
	misc	Merge branch 'master' of github.com:JoelKlint/twitter-geolocation-tra…	3 months ago
	startup	Fixa en bugg med relative path i startup/init.sh	3 months ago
	templates	Changed structure	3 months ago
	web	Fixed visualisation	2 months ago
	.env	Started tracking dependencies	3 months ago
	.gitignore	Added script to calculate holes in our capture of data	25 days ago
	README.md	Added table and code to filter possible locations	3 months ago
	calc_holes_in_db_data.py	Added script to calculate holes in our capture of data	25 days ago
	create_bounding_box_matrixes.py	Fixed accuracy, though to slow to be usefull	2 months ago
	detect_language.py	Added a language detector and hopefully fixed the float problem. The …	3 months ago
	geolocating-twitter-users.pdf	Added the report	24 days ago
	geonames_api_application.py	Removed some commented code	a month ago
	kartläggning.txt	Created script for exploiting geolocation functionality of google maps	3 months ago
	maps-geocode.js	Minor proposal	29 days ago
	parse_countries.py	Pushing latest	3 months ago
	place_with_contains.py	Removed some commented code	a month ago
	preprocess_user_location_in_db.py	Added a script which uses geonames to connect a user with a geoname e…	2 months ago
	requirements.txt	Updated dependencies	2 months ago
	run_algorithm.py	Removed some commented code	a month ago
	shell_arguments.py	Added verbosity support for database class	3 months ago
	tweepy_stream_listener.py	Removed some commented code	a month ago
	twitter_rest.py	Removed some commented code	a month ago
	twitter_stream.py	Removed some commented code	a month ago

 README.md

Todo

• Thread twitter save?
• Webbinterface

How to setup environment

1. Set these environment variables

TWITTER_CONSUMER_KEY
TWITTER_CONSUMER_SECRET
TWITTER_ACCESS_TOKEN
TWITTER_ACCESS_TOKEN_SECRET

2. Install PostgreSQL
3. Setup database

$ bash database/setup.sh

4. Install python dependencies

$ pip3 install virtualenv && virtualenv env && source env/bin/activate && pip3 install -r requirements.txt

5. Start datamining using your filters

bash /startup/init.sh start <filter 1> <filter 2> ...

6. Check which filters actually allowed by twitter

bash /startup/init.sh status

7. When done, stop the processes with.

bash /startup/init.sh stop

Antaganden Twitters API

• retweeted_id --> tweeten är en retweet
• in_reply_to_user_id --> någon har blivit mentioned
• in_reply_to_status_id --> tweeten är ett svar på en annan tweet
• original_tweet_retweet_count --> finns bara på kommenterade retweets
• En retweet är en "ren" retweet om attributet "original_tweet_retweet_count" inte är null
• En retweet är en "kommenterad" retweet om attributet "original_tweet_retweet_count" är null

Statements för att hämta ut statistik

SELECT count(*) FROM tweets WHERE retweeted_id IS NOT NULL AND in_reply_to_status_id IS NOT NULL AND in_reply_to_user_id IS NOT NULL;

SELECT count() FROM tweets WHERE retweeted_id IS NOT NULL AND in_reply_to_status_id IS NOT NULL AND in_reply_to_user_id IS NULL; SELECT count() FROM tweets WHERE retweeted_id IS NOT NULL AND in_reply_to_status_id IS NULL AND in_reply_to_user_id IS NOT NULL; SELECT count(*) FROM tweets WHERE retweeted_id IS NOT NULL AND in_reply_to_status_id IS NULL AND in_reply_to_user_id IS NULL;

SELECT count() FROM tweets WHERE retweeted_id IS NULL AND in_reply_to_status_id IS NOT NULL AND in_reply_to_user_id IS NOT NULL; SELECT count() FROM tweets WHERE retweeted_id IS NULL AND in_reply_to_status_id IS NOT NULL AND in_reply_to_user_id IS NULL; SELECT count() FROM tweets WHERE retweeted_id IS NULL AND in_reply_to_status_id IS NULL AND in_reply_to_user_id IS NOT NULL; SELECT count() FROM tweets WHERE retweeted_id IS NULL AND in_reply_to_status_id IS NULL AND in_reply_to_user_id IS NULL;

SELECT ALL FILTERED LOCATIONS:

select user_location, name, ratio, country_code from users inner join filtered_user_locations using (user_id) inner join geonames using(geonameid);