awesome-public-datasets/README.rst

463 lines
26 KiB
ReStructuredText
Raw Normal View History

2014-11-21 17:10:09 +08:00
Awesome Public Datasets
=======================
2015-08-08 00:30:45 +08:00
.. image:: https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg
:alt: Awesome
:target: https://github.com/sindresorhus/awesome
2014-12-21 15:38:35 +08:00
`This list of public data sources <https://github.com/caesar0301/awesome-public-datasets>`_
2015-08-09 01:19:50 +08:00
are collected and tidied from blogs, answers, and user reponses.
2014-12-21 15:38:35 +08:00
Most of the data sets listed below are free, however, some are not.
Other amazingly awesome lists can be found in the
`awesome-awesomeness <https://github.com/bayandin/awesome-awesomeness>`_ and
2015-04-28 09:54:10 +08:00
`sindresorhus's awesome <https://github.com/sindresorhus/awesome>`_ list.
2014-12-05 18:37:43 +08:00
2014-12-26 22:12:33 +08:00
Agriculture
------------
2014-12-27 00:27:06 +08:00
* `U.S. Department of Agriculture's PLANTS Database <http://www.plants.usda.gov/dl_all.html>`_
2014-12-26 22:12:33 +08:00
Biology
2014-11-21 17:10:09 +08:00
-------
2014-12-27 00:27:06 +08:00
* `1000 Genomes <http://www.1000genomes.org/data>`_
* `Collaborative Research in Computational Neuroscience (CRCNS) <http://crcns.org/data-sets>`_
* `Gene Expression Omnibus (GEO) <http://www.ncbi.nlm.nih.gov/geo/>`_
* `Human Microbiome Project (HMP) <http://www.hmpdacc.org/reference_genomes/reference_genomes.php>`_
* `American Gut (Microbiome Project) <https://github.com/biocore/American-Gut>`_
2014-12-27 00:27:06 +08:00
* `ICOS PSP Benchmark <http://www.infobiotic.net/PSPbenchmarks/>`_
* `MIT Cancer Genomics Data <http://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi>`_
2014-12-27 00:34:38 +08:00
* `NIH Microarray data (FTP) <http://bit.do/VVW6>`_
2015-07-31 00:31:56 +08:00
* `OpenSNP genotypes data <https://opensnp.org/>`_
2014-12-27 00:27:06 +08:00
* `Protein Data Bank <http://pdb.org/>`_
* `PubChem Project <https://pubchem.ncbi.nlm.nih.gov/>`_
* `PubGene (now Coremine Medical) <http://www.pubgene.org/>`_
* `Stanford Microarray Data <http://smd.stanford.edu/>`_
2014-12-27 00:34:38 +08:00
* `The Personal Genome Project <http://www.personalgenomes.org/>`_ or `PGP <https://my.pgp-hms.org/public_genetic_data>`_
2014-12-27 00:27:06 +08:00
* `UCSC Public Data <http://hgdownload.soe.ucsc.edu/downloads.html>`_
* `UniGene <http://www.ncbi.nlm.nih.gov/unigene>`_
2014-12-26 22:12:33 +08:00
Climate/Weather
---------------
2014-12-27 00:27:06 +08:00
* `Australian Weather <http://www.bom.gov.au/climate/dwo/>`_
* `Brazilian Weather - Historical data (In Portuguese) <http://sinda.crn2.inpe.br/PCD/SITE/novo/site/>`_
2014-12-27 00:27:06 +08:00
* `Canadian Meteorological Centre <https://weather.gc.ca/grib/index_e.html>`_
2015-01-31 17:18:37 +08:00
* `Climate Data from UEA (updated monthly) <http://www.cru.uea.ac.uk/cru/data/temperature/#datter and ftp://ftp.cmdl.noaa.gov/>`_
2014-12-27 00:27:06 +08:00
* `Global Climate Data Since 1929 <http://www.tutiempo.net/en/Climate>`_
* `NASA Global Imagery Browse Services <https://wiki.earthdata.nasa.gov/display/GIBS>`_
2014-12-27 00:27:06 +08:00
* `NOAA Bering Sea Climate <http://www.beringclimate.noaa.gov/>`_
* `NOAA Climate Datasets <http://ncdc.noaa.gov/data-access/quick-links>`_
* `NOAA Realtime Weather Models <http://www.ncdc.noaa.gov/data-access/model-data/model-datasets/numerical-weather-prediction>`_
2015-04-21 14:54:28 +08:00
* `The World Bank Open Data Resources for Climate Change <http://data.worldbank.org/developers/climate-data-api>`_
* `UEA Climatic Research Unit <http://www.cru.uea.ac.uk/data>`_
2014-12-27 00:27:06 +08:00
* `WU Historical Weather Worldwide <http://www.wunderground.com/history/index.html>`_
2014-11-21 17:10:09 +08:00
2014-12-05 18:37:43 +08:00
2014-12-26 22:12:33 +08:00
Complex Networks
----------------
2014-12-27 00:27:06 +08:00
* `CrossRef DOI URLs <https://archive.org/details/doi-urls>`_
* `DBLP Citation dataset <https://kdl.cs.umass.edu/display/public/DBLP>`_
* `NBER Patent Citations <http://nber.org/patents/>`_
* `NIST complex networks data collection <http://math.nist.gov/~RPozo/complex_datasets.html>`_
2015-04-02 05:36:57 +08:00
* `Small Network Data <http://www-personal.umich.edu/~mejn/netdata/>`_
* `UCI Network Data Repository <https://networkdata.ics.uci.edu/resources.php>`_
2014-12-27 00:27:06 +08:00
* `Protein-protein interaction network <http://vlado.fmf.uni-lj.si/pub/networks/data/bio/Yeast/Yeast.htm>`_
* `PyPI and Maven Dependency Network <http://ogirardot.wordpress.com/2013/01/31/sharing-pypimaven-dependency-data/>`_
* `Scopus Citation Database <http://www.elsevier.com/online-tools/scopus>`_
* `Stanford GraphBase (Steven Skiena) <http://www3.cs.stonybrook.edu/~algorith/implement/graphbase/implement.shtml>`_
* `Stanford Large Network Dataset Collection <http://snap.stanford.edu/data/>`_
* `The Koblenz Network Collection <http://konect.uni-koblenz.de/>`_
* `The Laboratory for Web Algorithmics (UNIMI) <http://law.di.unimi.it/datasets.php>`_
2015-04-14 15:51:04 +08:00
* `The Nexus Network Repository <http://nexus.igraph.org/>`_
2014-12-27 00:27:06 +08:00
* `UCI Network Data Repository <http://networkdata.ics.uci.edu/resources.php>`_
* `UFL sparse matrix collection <http://www.cise.ufl.edu/research/sparse/matrices/>`_
* `WSU Graph Database <http://www.eecs.wsu.edu/mgd/gdb.html>`_
2014-12-26 22:12:33 +08:00
Computer Networks
-----------------
2015-01-31 17:18:37 +08:00
* `3.5B Web Pages from CommonCraw 2012 <http://www.bigdatanews.com/profiles/blogs/big-data-set-3-5-billion-web-pages-made-available-for-all-of-us>`_
* `53.5B Web clicks of 100K users in Indiana Univ. <http://cnets.indiana.edu/groups/nan/webtraffic/click-dataset>`_
* `CAIDA Internet Datasets <http://www.caida.org/data/overview/>`_
* `ClueWeb09 - 1B web pages <http://lemurproject.org/clueweb09/>`_
* `ClueWeb12 - 733M web pages <http://lemurproject.org/clueweb12/>`_
* `CommonCrawl Web Data over 7 years <http://commoncrawl.org/the-data/get-started/>`_
* `CRAWDAD Wireless datasets from Dartmouth Univ. <http://crawdad.cs.dartmouth.edu/>`_
2015-04-01 05:48:15 +08:00
* `Criteo click-through data <http://labs.criteo.com/2015/03/criteo-releses-its-new-dataset/>`_
2015-01-31 17:18:37 +08:00
* `Open Mobile Data by MobiPerf <https://console.developers.google.com/storage/openmobiledata_public/>`_
* `UCSD Network Telescope, IPv4 /8 net <http://www.caida.org/projects/network_telescope/>`_
2014-12-26 22:12:33 +08:00
Data Challenges
---------------
2014-12-27 00:27:06 +08:00
* `Challenges in Machine Learning <http://www.chalearn.org/>`_
2015-04-01 13:09:03 +08:00
* `D4D Challenge of Orange <http://www.d4d.orange.com/en/home>`_
2014-12-27 00:27:06 +08:00
* `DrivenData Competitions for Social Good <http://www.drivendata.org/>`_
* `ICWSM Data Challenge (since 2009) <http://icwsm.cs.umbc.edu/>`_
* `Kaggle Competition Data <http://www.kaggle.com/>`_
* `KDD Cup by Tencent 2012 <https://www.kddcup2012.org/>`_
* `Localytics Data Visualization Challenge <https://github.com/localytics/data-viz-challenge>`_
* `Netflix Prize <http://www.netflixprize.com/leaderboard>`_
2015-04-15 21:19:21 +08:00
* `Space Apps Challenge <https://www.spaceappschallenge.org>`_
* `Telecom Italia Big Data Challenge <https://dandelion.eu/datamine/open-big-data/>`_
2014-12-27 00:27:06 +08:00
* `Yelp Dataset Challenge <http://www.yelp.com/dataset_challenge>`_
2014-12-26 22:12:33 +08:00
2014-11-21 17:10:09 +08:00
Economics
---------
2015-01-31 17:18:37 +08:00
* `American Economic Ass (AEA) <http://www.aeaweb.org/RFE/toc.php?show=complete>`_
2014-12-27 00:27:06 +08:00
* `EconData from UMD <http://inforumweb.umd.edu/econdata/econdata.html>`_
* `Internet Product Code Database <http://www.upcdatabase.com/>`_
2014-11-21 17:10:09 +08:00
2014-12-05 18:37:43 +08:00
2014-12-02 02:52:10 +08:00
Energy
------
2014-12-27 00:27:06 +08:00
* `AMPds <http://ampds.org/>`_
* `BLUEd <http://nilm.cmubi.org/>`_
* `COMBED <http://combed.github.io/>`_
* `Dataport <https://dataport.pecanstreet.org/>`_
* `ECO <http://www.vs.inf.ethz.ch/res/show.html?what=eco-data>`_
* `EIA <http://www.eia.gov/electricity/data/eia923/>`_
* `HFED <http://hfed.github.io/>`_
* `iAWE <http://iawe.github.io/>`_
* `Plaid <http://plaidplug.com/>`_
* `REDD <http://redd.csail.mit.edu/>`_
* `UK-Dale <http://www.doc.ic.ac.uk/~dk3810/data/>`_
2014-12-02 02:52:10 +08:00
2014-11-21 17:10:09 +08:00
Finance
-------
2014-12-27 00:27:06 +08:00
* `CBOE Futures Exchange <http://cfe.cboe.com/Data/>`_
* `Google Finance <https://www.google.com/finance>`_
* `Google Trends <http://www.google.com/trends?q=google&ctab=0&geo=all&date=all&sort=0>`_
* `NASDAQ <https://data.nasdaq.com/>`_
* `OANDA <http://www.oanda.com/>`_
2014-12-27 00:34:38 +08:00
* `OSU Financial data <http://fisher.osu.edu/fin/fdf/osudata.htm>`_
2014-12-27 00:27:06 +08:00
* `Quandl <http://www.quandl.com/>`_
* `St Louis Federal <http://research.stlouisfed.org/fred2/>`_
* `Yahoo Finance <http://finance.yahoo.com/>`_
2014-11-21 17:10:09 +08:00
2015-04-28 01:21:18 +08:00
Geology
-------
* `USGS Earthquake Archives <http://earthquake.usgs.gov/earthquakes/search/>`_
* `Smithsonian Institution Global Volcano and Eruption Database <http://www.volcano.si.edu>`_
2014-12-05 18:37:43 +08:00
GeoSpace/GIS
2014-12-26 22:12:33 +08:00
------------
2014-11-21 17:10:09 +08:00
2015-01-31 17:18:37 +08:00
* `BODC - marine data of ~22K vars <http://www.bodc.ac.uk/data/where_to_find_data/>`_
* `Cambridge, MA, US, GIS data on GitHub <http://cambridgegis.github.io/gisdata.html>`_
2015-01-31 17:18:37 +08:00
* `EOSDIS - NASA's earth observing system data <http://sedac.ciesin.columbia.edu/data/sets/browse>`_
* `Factual Global Location Data <http://www.factual.com/>`_
* `Geo Spatial Data from ASU <http://geodacenter.asu.edu/datalist/>`_
* `GeoNames Worldwide <http://www.geonames.org/>`_
* `Global Administrative Areas Database (GADM) <http://www.gadm.org/>`_
* `Landsat 8 on AWS <https://aws.amazon.com/public-data-sets/landsat/>`_
2015-01-31 17:18:37 +08:00
* `Natural Earth - vectors and rasters of the world <http://www.naturalearthdata.com/>`_
2015-07-10 03:49:48 +08:00
* `OpenStreetMap (OSM) <http://wiki.openstreetmap.org/wiki/Downloading_data>`_
2015-01-31 17:18:37 +08:00
* `TIGER/Line - U.S. boundaries and roads <http://www.census.gov/geo/maps-data/data/tiger-line.html>`_
* `TwoFishes - Foursquare's coarse geocoder <https://github.com/foursquare/twofishes>`_
* `TZ Timezones shapfiles <http://efele.net/maps/tz/world/>`_
2015-04-02 20:04:02 +08:00
* `World countries in multiple formats <https://github.com/mledoze/countries>`_
2015-04-28 09:54:10 +08:00
* `List of all countries in all languages <https://github.com/umpirsky/country-list>`_
2015-04-16 11:53:43 +08:00
* `OpenAddresses <http://openaddresses.io/>`_
2014-11-21 17:10:09 +08:00
Government
----------
2015-04-21 02:57:00 +08:00
* `Austin, TX, US <https://data.austintexas.gov/>`_
2015-01-31 17:18:37 +08:00
* `Australia (abs.gov.au) <http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/3301.02009?OpenDocument>`_
* `Australia (data.gov.au) <https://data.gov.au/>`_
* `Austria (data.gv.at) <https://www.data.gv.at/>`_
* `Brazil <http://dados.gov.br/dataset>`_
* `Cambridge, MA, US <https://data.cambridgema.gov/>`_
2014-12-27 00:27:06 +08:00
* `Canada <http://www.data.gc.ca/default.asp?lang=En&n=5BCD274E-1>`_
* `Chicago <https://data.cityofchicago.org/>`_
* `Dallas Open Data <https://www.dallasopendata.com/>`_
* `Denver Open Data <http://data.denvergov.org//>`_
2015-08-16 10:26:04 +08:00
* `Durham, NC Open Data <https://opendurham.nc.gov/explore/>`_
2015-04-20 16:08:29 +08:00
* `England LGInform <http://lginform.local.gov.uk/>`_
2014-12-27 00:27:06 +08:00
* `EuroStat <http://ec.europa.eu/eurostat/data/database>`_
* `FedStats <http://www.fedstats.gov/cgi-bin/A2Z.cgi>`_
2015-04-19 10:03:34 +08:00
* `Finland <https://www.opendata.fi/en>`_
* `France <https://www.data.gouv.fr/en/datasets/>`_
2014-12-27 00:27:06 +08:00
* `Germany <https://www-genesis.destatis.de/genesis/online>`_
* `Glasgow, Scotland, UK <http://data.glasgow.gov.uk/>`_
* `Guardian world governments <http://www.guardian.co.uk/world-government-data>`_
2015-06-12 04:53:07 +08:00
* `Houston Open Data <http://data.ohouston.org>`_
2015-02-23 20:28:38 +08:00
* `Indian Government Data <http://www.data.gov.in>`_
* `Indonesian Data Portal <http://data.go.id/>`_
2015-01-30 10:43:36 +08:00
* `London Datastore, UK <http://data.london.gov.uk/dataset>`_
2015-06-08 02:16:06 +08:00
* `Los Angeles Open Data <https://data.lacity.org/>`_
2015-01-30 10:43:36 +08:00
* `MassGIS, Massachusetts, U.S. <http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/>`_
* `Mexico <http://catalogo.datos.gob.mx/dataset>`_
2014-12-27 00:27:06 +08:00
* `Netherlands <https://data.overheid.nl/>`_
* `New Zealand <http://www.stats.govt.nz/browse_for_stats.aspx>`_
* `NYC betanyc <http://betanyc.us/>`_
* `NYC Open Data <http://nycplatform.socrata.com/>`_
* `OECD <http://www.oecd.org/document/0,3746,en_2649_201185_46462759_1_1_1_1,00.html>`_
* `Oklahoma <https://data.ok.gov/>`_
2014-12-27 00:27:06 +08:00
* `Open Government Data (OGD) Platform India <http://www.data.gov.in/>`_
* `Rio de Janeiro, Brazil <http://data.rio.rj.gov.br/>`_
2015-04-21 14:12:07 +08:00
* `Romania <http://data.gov.ro/>`_
2014-12-27 00:27:06 +08:00
* `San Francisco Data sets <http://datasf.org/>`_
* `Seattle <https://data.seattle.gov/>`_
2015-08-03 10:44:30 +08:00
* `Singapore Government Data <http://www.data.gov.sg/>`_
2014-12-27 00:27:06 +08:00
* `South Africa <http://beta2.statssa.gov.za/>`_
2015-04-21 00:44:31 +08:00
* `Switzerland <http://www.opendata.admin.ch/>`_
2014-12-27 00:27:06 +08:00
* `The World Bank <http://wdronline.worldbank.org/>`_
* `Texas Open Data <https://data.texas.gov/>`_
* `Puerto Rico Government <https://data.pr.gov//>`_
2014-12-27 00:27:06 +08:00
* `U.K. Government Data <http://data.gov.uk/data>`_
2015-04-21 20:53:36 +08:00
* `Uruguay <https://catalogodatos.gub.uy/>`_
2014-12-27 00:27:06 +08:00
* `U.S. American Community Survey <http://www.census.gov/acs/www/data_documentation/data_release_info/>`_
* `U.S. CDC Public Health datasets <http://www.cdc.gov/nchs/data_access/ftp_data.htm>`_
* `U.S. Census Bureau <http://www.census.gov/data.html>`_
2015-04-20 16:08:29 +08:00
* `U.S. National Center for Education Statistics (NCES) <http://nces.ed.gov/>`_
2014-12-27 00:27:06 +08:00
* `U.S. Department of Housing and Urban Development (HUD) <http://www.huduser.org/portal/datasets/pdrdatas.html>`_
* `U.S. Federal Government Agencies <http://www.data.gov/metric>`_
* `U.S. Federal Government Data Catalog <http://catalog.data.gov/dataset>`_
* `U.S. Food and Drug Administration (FDA) <https://open.fda.gov/index.html>`_
* `U.S. Open Government <http://www.data.gov/open-gov/>`_
2015-04-22 23:58:36 +08:00
* `UK 2011 Census Open Atlas Project <http://www.alex-singleton.com/r/2013/02/05/2011-census-open-atlas-project/>`_
2014-12-27 00:27:06 +08:00
* `United Nations <http://data.un.org/>`_
2015-07-09 15:55:53 +08:00
* `Vancouver, BC Open Data Catalog <http://data.vancouver.ca/datacatalogue/>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Healthcare
----------
2015-01-31 17:18:37 +08:00
* `EHDP Large Health Data Sets <http://www.ehdp.com/vitalnet/datasets.htm>`_
* `Gapminder World, demographic databases <http://www.gapminder.org/data/>`_
* `Medicare Coverage Database (MCD), U.S. <http://www.cms.gov/medicare-coverage-database/>`_
* `Medicare Data Engine of medicare.gov Data <https://data.medicare.gov/>`_
2014-12-27 00:27:06 +08:00
* `Medicare Data File <http://go.cms.gov/19xxPN4>`_
* `Number of Ebola Cases and Deaths in Affected Countries (2014) <https://data.hdx.rwlabs.org/dataset/ebola-cases-2014>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Image Processing
----------------
2014-11-21 17:10:09 +08:00
* `10k US Adult Faces Database <http://wilmabainbridge.com/facememorability2.html>`_
2015-08-20 17:36:34 +08:00
* `2GB of Photos of Cats (Original down - 20Agst2015) <http://137.189.35.203/WebUI/CatDatabase/catData.html>`_ or `Archive version <https://web.archive.org/web/20150520175645/http://137.189.35.203/WebUI/CatDatabase/catData.html>`_
2015-08-20 16:23:43 +08:00
* `Stanford Dogs Dataset <http://vision.stanford.edu/aditya86/ImageNetDogs/>`_
* `The Oxford-IIIT Pet Dataset <http://www.robots.ox.ac.uk/~vgg/data/pets/>`_
* `Animals with attributes <http://attributes.kyb.tuebingen.mpg.de/>`_
* `Affective Image Classification <http://www.imageemotion.org/>`_
2015-01-31 17:18:37 +08:00
* `Face Recognition Benchmark <http://www.face-rec.org/databases/>`_
* `ImageNet (in WordNet hierarchy) <http://www.image-net.org/>`_
* `International Affective Picture System, UFL <http://csea.phhp.ufl.edu/media/iapsmessage.html>`_
* `Massive Visual Memory Stimuli, MIT <http://cvcl.mit.edu/MM/stimuli.html>`_
* `SUN database, MIT <http://groups.csail.mit.edu/vision/SUN/hierarchy.html>`_
2015-04-23 13:26:33 +08:00
* `YouTube Faces Database <http://www.cs.tau.ac.il/~wolf/ytfaces/>`_
* `Indoor Scene Recognition <http://web.mit.edu/torralba/www/indoor.html>`_
2014-11-21 17:10:09 +08:00
Machine Learning
----------------
2015-01-31 17:18:37 +08:00
* `Delve Datasets for classification and regression (Univ. of Toronto) <http://www.cs.toronto.edu/~delve/data/datasets.html>`_
* `Discogs Monthly Data <http://www.discogs.com/data/>`_
* `eBay Online Auctions (2012) <http://www.modelingonlineauctions.com/datasets>`_
* `IMDb Database <http://www.imdb.com/interfaces>`_
* `Keel Repository for classification, regression and time series <http://sci2s.ugr.es/keel/datasets.php>`_
* `Lending Club Loan Data <https://www.lendingclub.com/info/download-data.action>`_
* `Machine Learning Data Set Repository <http://mldata.org/>`_
* `Million Song Dataset <http://labrosa.ee.columbia.edu/millionsong/>`_
* `More Song Datasets <http://labrosa.ee.columbia.edu/millionsong/pages/additional-datasets>`_
* `MovieLens Data Sets <http://grouplens.org/datasets/movielens/>`_
* `RDataMining - "R and Data Mining" ebook data <http://www.rdatamining.com/data>`_
2015-01-31 17:18:37 +08:00
* `Registered Meteorites on Earth <http://www.analyticbridge.com/profiles/blogs/registered-meteorites-that-has-impacted-on-earth-visualized>`_
* `Restaurants Health Score Data in San Francisco <http://missionlocal.org/san-francisco-restaurant-health-inspections/>`_
* `UCI Machine Learning Repository <http://archive.ics.uci.edu/ml/>`_
* `Yahoo! Ratings and Classification Data <http://webscope.sandbox.yahoo.com/catalog.php?datatype=r>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Museums
-------
2014-12-27 00:27:06 +08:00
* `Cooper-Hewitt's Collection Database <https://github.com/cooperhewitt/collection>`_
* `Minneapolis Institute of Arts metadata <https://github.com/artsmia/collection>`_
* `Tate Collection metadata <https://github.com/tategallery/collection>`_
* `The Getty vocabularies <http://vocab.getty.edu>`_
* `Rijksmuseum Historical Art Collection <https://www.rijksmuseum.nl/en/api>`_
2014-12-26 22:12:33 +08:00
2014-11-21 17:10:09 +08:00
Natural Language
----------------
2015-02-26 11:56:02 +08:00
* `Blogger Corpus <http://u.cs.biu.ac.il/~koppel/BlogCorpus.htm>`_
2015-01-31 17:18:37 +08:00
* `ClueWeb09 FACC <http://lemurproject.org/clueweb09/FACC1/>`_
* `ClueWeb12 FACC <http://lemurproject.org/clueweb12/FACC1/>`_
2015-01-31 17:21:26 +08:00
* `DBpedia - 4.58M things with 583M facts <http://wiki.dbpedia.org/Datasets>`_
2015-01-31 17:18:37 +08:00
* `Flickr Personal Taxonomies <http://www.isi.edu/~lerman/downloads/flickr/flickr_taxonomies.html>`_
* `Google Books Ngrams (2.2TB) <http://aws.amazon.com/datasets/8172056142375670>`_
* `Google Web 5gram (1TB, 2006) <https://catalog.ldc.upenn.edu/LDC2006T13>`_
* `Gutenberg eBooks List <http://www.gutenberg.org/wiki/Gutenberg:Offline_Catalogs>`_
* `Hansards text chunks of Canadian Parliament <http://www.isi.edu/natural-language/download/hansard/>`_
* `Machine Translation of European languages <http://statmt.org/wmt11/translation-task.html#download>`_
* `SMS Spam Collection in English <http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/>`_
* `SaudiNewsNet Collection of Saudi Newspaper Articles (Arabic, 30K articles) <https://github.com/ParallelMazen/SaudiNewsNet>`_
2015-01-31 17:18:37 +08:00
* `USENET postings corpus of 2005~2011 <http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus.download.html>`_
* `Wikidata - Wikipedia databases <https://www.wikidata.org/wiki/Wikidata:Database_download>`_
* `Wikipedia Links data - 40 Million Entities in Context <https://code.google.com/p/wiki-links/downloads/list>`_
* `WordNet databases and tools <http://wordnet.princeton.edu/wordnet/download/>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Physics
-------
2014-11-21 17:10:09 +08:00
2015-01-31 17:18:37 +08:00
* `CERN Open Data Portal <http://opendata.cern.ch/>`_
* `NSSDC (NASA) data of 550 space spacecraft <http://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html>`_
* `NASA Exoplanet Archive <http://exoplanetarchive.ipac.caltech.edu/>`_
* `Sloan Digital Sky Survey (SDSS) - Mapping the Universe <http://www.sdss.org/>`_
2014-11-21 17:10:09 +08:00
2015-05-20 06:27:51 +08:00
Psychology/Cognition
--------------
2015-05-21 06:14:15 +08:00
* `OSU Cognitive Modeling Repository Datasets <http://www.cmr.osu.edu/browse/datasets>`_
2015-05-20 06:27:51 +08:00
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Public Domains
--------------
2014-12-27 00:27:06 +08:00
* `Amazon <http://aws.amazon.com/datasets>`_
* `Archive.org Datasets <https://archive.org/details/datasets>`_
* `CMU JASA data archive <http://lib.stat.cmu.edu/jasadata/>`_
* `CMU StatLab collections <http://lib.stat.cmu.edu/datasets/>`_
* `Data360 <http://www.data360.org/index.aspx>`_
* `Datamob.org <http://datamob.org/datasets>`_
* `Google <http://www.google.com/publicdata/directory>`_
* `Infochimps <http://www.infochimps.com/>`_
* `KDNuggets Data Collections <http://www.kdnuggets.com/datasets/index.html>`_
* `Microsoft Azure Data Market Free DataSets <http://datamarket.azure.com/browse/data?price=free>`_
2014-12-27 00:27:06 +08:00
* `Numbray <http://numbrary.com/>`_
* `Reddit Datasets <http://www.reddit.com/r/datasets>`_
* `RevolutionAnalytics Collection <http://www.revolutionanalytics.com/subscriptions/datasets/>`_
* `Sample R data sets <http://stat.ethz.ch/R-manual/R-patched/library/datasets/html/00Index.html>`_
* `Stats4Stem R data sets <http://www.stats4stem.org/data-sets.html>`_
* `StatSci.org <http://www.statsci.org/datasets.html>`_
* `The Washington Post List <http://www.washingtonpost.com/wp-srv/metro/data/datapost.html>`_
* `UCLA SOCR data collection <http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data>`_
* `UFO Reports <http://www.nuforc.org/webreports.html>`_
* `Wikileaks 911 pager intercepts <http://911.wikileaks.org/files/index.html>`_
* `Yahoo Webscope <http://webscope.sandbox.yahoo.com/catalog.php>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Search Engines
--------------
2015-01-31 17:18:37 +08:00
* `Academic Torrents of data sharing from UMB <http://academictorrents.com/>`_
* `Archive-it from Internet Archive <https://www.archive-it.org/explore?show=Collections>`_
* `Datahub.io <http://datahub.io/dataset>`_
2015-01-06 16:17:58 +08:00
* `DataMarket (Qlik) <https://datamarket.com/data/list/?q=all>`_
2015-01-31 17:18:37 +08:00
* `Freebase.com of people, places, and things <http://www.freebase.com/>`_
* `Harvard Dataverse Network of scientific data <http://thedata.harvard.edu/dvn/>`_
* `ICPSR (UMICH) <http://www.icpsr.umich.edu/icpsrweb/ICPSR/index.jsp>`_
* `Open Data Certificates (beta) <https://certificates.theodi.org/datasets>`_
2015-01-31 17:18:37 +08:00
* `Statista.com - statistics and Studies <http://www.statista.com/>`_
2014-11-21 17:10:09 +08:00
2015-08-16 11:38:59 +08:00
Social Networks
---------------
2015-08-16 11:41:36 +08:00
* `72 hours #gamergate scrape <http://waxy.org/random/misc/gamergate_tweets.csv>`_
* `Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape <http://www.archive.org/details/twitter_cikm_2010>`_
* `May 2011 Calufa Twitter Scrape <http://www.archive.org/details/2011-05-calufa-twitter-sql>`_
* `Network Twitter Data <http://snap.stanford.edu/data/higgs-twitter.html>`_
* `Social Twitter Data <http://snap.stanford.edu/data/egonets-Twitter.html>`_
* `Twitter Data for Sentiment Analysis <http://help.sentiment140.com/for-students/>`_
2014-11-21 17:10:09 +08:00
Social Sciences
---------------
2015-01-31 17:18:37 +08:00
* `Ancestry.com Forum Dataset over 10 years <http://www.cs.cmu.edu/~jelsas/data/ancestry.com/>`_
* `CMU Enron Email of 150 users <http://www.cs.cmu.edu/~enron/>`_
2015-04-22 02:03:36 +08:00
* `EDRM Enron EMail of 151 users, hosted on S3 <https://aws.amazon.com/datasets/917205>`_
2015-01-31 17:18:37 +08:00
* `Facebook Data Scrape (2005) <https://archive.org/details/oxford-2005-facebook-matrix>`_
* `Facebook Social Networks from LAW (since 2007) <http://law.di.unimi.it/datasets.php>`_
2015-08-03 10:44:30 +08:00
* `FBI Hate Crime 2013 - aggregated data <https://github.com/emorisse/FBI-Hate-Crime-Statistics/tree/master/2013>`_
2015-01-31 17:18:37 +08:00
* `Foursquare Social Network in 2010, 2011 <http://www.public.asu.edu/~hgao16/dataset.html>`_
* `Foursquare from UMN/Sarwat (2013) <https://archive.org/details/201309_foursquare_dataset_umn>`_
* `General Social Survey (GSS) since 1972 <http://www3.norc.org/GSS+Website/>`_
* `GetGlue - users rating TV shows <http://bit.ly/1aL8XS0>`_
* `GitHub Collaboration Archive <http://www.githubarchive.org/>`_
* `MIT Reality Mining Dataset <http://realitycommons.media.mit.edu/realitymining.html>`_
2015-01-31 17:18:37 +08:00
* `Mobile Social Networks from UMASS <https://kdl.cs.umass.edu/display/public/Mobile+Social+Networks>`_
* `PewResearch Internet Survey Project <http://www.pewinternet.org/datasets/pages/2/>`_
2015-07-14 17:44:44 +08:00
* `Reddit Comments <https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/>`_
2015-01-31 17:18:37 +08:00
* `SourceForge.net Research Data <http://www.nd.edu/~oss/Data/data.html>`_
* `StackExchange Data Explorer <http://data.stackexchange.com/help>`_
2015-04-28 09:54:10 +08:00
* `Titanic Survival Data Set <https://github.com/caesar0301/awesome-public-datasets/tree/master/Datasets>`_
2015-08-16 10:26:04 +08:00
* `Texas Inmates Executed Since 1984 <http://www.tdcj.state.tx.us/death_row/dr_executed_offenders.html>`_
2015-01-31 17:18:37 +08:00
* `Twitter Graph of entire Twitter site <http://an.kaist.ac.kr/traces/WWW2010.html>`_
* `UCB's Archive of Social Science Data (D-Lab) <http://ucdata.berkeley.edu/>`_
* `UCLA Social Sciences Data Archive <http://dataarchives.ss.ucla.edu/Home.DataPortals.htm>`_
* `UNIMI/LAW Social Network Datasets <http://law.di.unimi.it/datasets.php>`_
* `Universities Worldwide <http://univ.cc/>`_
* `UPJOHN for Labor Employment Research <http://www.upjohn.org/erdc/erdc.html>`_
* `Yahoo! Graph and Social Data <http://webscope.sandbox.yahoo.com/catalog.php?datatype=g>`_
* `Youtube Video Social Graph in 2007,2008 <http://netsg.cs.sfu.ca/youtubedata/>`_
2015-03-06 17:26:38 +08:00
* `Google Scholar citation relations <http://www3.cs.stonybrook.edu/~leman/data/gscholar.db>`_
* `Political Polarity Data <http://www3.cs.stonybrook.edu/~leman/data/14-icwsm-political-polarity-data.zip>`_
2015-08-01 08:44:25 +08:00
* `GDELT Global Events Database <http://gdeltproject.org/data.html>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Sports
------
2014-12-08 17:42:02 +08:00
2015-01-31 17:18:37 +08:00
* `Betfair Historical Exchange Data <http://data.betfair.com/>`_
* `Cricsheet Matches (cricket) <http://cricsheet.org/>`_
2015-01-31 17:18:37 +08:00
* `Ergast Formula 1, from 1950 up to date (API) <http://ergast.com/mrd/db>`_
* `Football/Soccer resouces (data and APIs) <http://www.jokecamp.com/blog/guide-to-football-and-soccer-data-and-apis/>`_
2015-01-31 17:18:37 +08:00
* `Lahman's Baseball Database <http://www.seanlahman.com/baseball-archive/statistics/>`_
* `Retrosheet Baseball Statistics <http://www.retrosheet.org/game.htm>`_
2014-12-26 22:12:33 +08:00
Time Series
-----------
2014-11-21 17:10:09 +08:00
2015-01-31 17:18:37 +08:00
* `Time Series Data Library (TSDL) from MU <https://datamarket.com/data/list/?q=provider:tsdl>`_
* `UC Riverside Time Series Dataset <http://www.cs.ucr.edu/~eamonn/time_series_data/>`_
2015-02-05 03:36:47 +08:00
* `Hard Drive Failure Rates <https://www.backblaze.com/hard-drive-test-data.html>`_
* `Heart Rate Time Series from MIT <http://ecg.mit.edu/time-series/>`_
2014-11-21 17:10:09 +08:00
2014-12-26 22:12:33 +08:00
Transportation
2014-11-21 17:10:09 +08:00
--------------
2015-01-31 17:18:37 +08:00
* `Airlines OD Data 1987-2008 <http://stat-computing.org/dataexpo/2009/the-data.html>`_
* `Bike Share Systems (BSS) collection <https://github.com/BetaNYC/Bike-Share-Data-Best-Practices/wiki/Bike-Share-Data-Systems>`_
* `Bay Area Bike Share Data <http://www.bayareabikeshare.com/datachallenge>`_
* `GeoLife GPS Trajectory from Microsoft Research <http://research.microsoft.com/en-us/downloads/b16d359d-d164-469e-9fd4-daa38f2b2e13/>`_
2015-01-31 17:18:37 +08:00
* `Hubway Million Rides in MA <http://hubwaydatachallenge.org/trip-history-data/>`_
* `Marine Traffic - ship tracks, port calls and more <https://www.marinetraffic.com/de/p/api-services>`_
* `NYC Taxi Trip Data 2013 (FOIA/FOILed) <https://archive.org/details/nycTaxiTripData2013>`_
* `OpenFlights - airport, airline and route data <http://openflights.org/data.html>`_
* `RITA Airline On-Time Performance data <http://www.transtats.bts.gov/Tables.asp?DB_ID=120>`_
* `RITA/BTS transport data collection (TranStat) <http://www.transtats.bts.gov/DataIndex.asp>`_
2015-01-31 17:18:37 +08:00
* `Transport for London (TFL) <http://www.tfl.gov.uk/info-for/open-data-users/our-feeds>`_
* `Travel Tracker Survey (TTS) for Chicago <http://www.cmap.illinois.gov/data/transportation/travel-tracker-survey>`_
* `U.S. Bureau of Transportation Statistics (BTS) <http://www.rita.dot.gov/bts/>`_
2015-03-20 10:41:24 +08:00
* `U.S. Domestic Flights 1990 to 2009 <http://academictorrents.com/details/a2ccf94bbb4af222bf8e69dad60a68a29f310d9a>`_
2015-01-31 17:18:37 +08:00
* `U.S. Freight Analysis Framework since 2007 <http://ops.fhwa.dot.gov/freight/freight_analysis/faf/index.htm>`_
2014-11-21 17:10:09 +08:00
Complementary Collections
-------------------------
2014-12-27 00:27:06 +08:00
* DataWrangling: `Some Datasets Available on the Web <http://www.datawrangling.com/some-datasets-available-on-the-web>`_
* Inside-r: `Finding Data on the Internet <http://www.inside-r.org/howto/finding-data-internet>`_
* Quora: `Where can I find large datasets open to the public? <http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public>`_
* RS.io: `100+ Interesting Data Sets for Statistics <http://rs.io/2014/05/29/list-of-data-sets.html>`_
2015-03-18 15:03:50 +08:00
* StaTrek: `Leveraging open data to understand urban lives <http://xiaming.me/posts/2014/10/23/leveraging-open-data-to-understand-urban-lives/>`_
2015-04-18 19:54:03 +08:00
* OpenDataMonitor: `An overview of available open data resources in Europe <http://opendatamonitor.eu>`_