Whether you\u2019re starting a new project or expanding an existing one, as a data scientist, you\u2019re always on the lookout for new material to explore. Knowing where to get data for data science projects can be challenging, and finding \u201cgood data\u201d can be even more difficult. In this article, we\u2019ll look at what makes \u201cgood data\u201d, what format that data might be in, where to find it, and what the next steps are. <\/p>\n\n\n\n
Firstly, we should consider how relevant<\/em> the dataset is to our work. You can stumble upon lots of datasets that overlap with your work in some way, but it can be difficult to decide which is the best one for you to put your effort into. In this scenario, we\u2019ll briefly explore some of the attributes of the data. <\/p>\n\n\n\n To start with, how consistent<\/em> is the dataset? Specifically, are there any missing values? Data might be missing for a variety of acceptable reasons, but it can also be a sign of selection bias<\/a> or other factors that might skew your results. Often, we can choose to either accept missing data or delete the records that contain it before we do our analysis, but knowing about missing data early in the process can help you make an informed decision to use that dataset or not. <\/p>\n\n\n\n Along with missing data, it\u2019s worth checking to see if any of the data is duplicated<\/em>. Duplicated data might be fine, but it might also signify a lack of consistency that could skew your results. Duplicated data might also reduce your confidence in the dataset as a whole, so it\u2019s important to consider when choosing your dataset. <\/p>\n\n\n\n Another aspect to consider for good data is timeliness<\/em>. The time over which the data was gathered is usually pertinent to the questions you want to answer when you start analyzing it. Checking if the data was collected in the timespan that you\u2019re interested in and considering the continuity of that timespan is helpful. <\/p>\n\n\n\n When you\u2019re starting your journey into data science and picking your first few datasets to play with, you don\u2019t need to worry about picking the perfect dataset \u2013 focus on the process and exploring instead. When you\u2019re ready to learn more about datasets and how to avoid common pitfalls, I recommend you watch this talk from Dr. Jodie Burchell \u2013 Garbage data in, garbage models out<\/em><\/a>.<\/p>\n\n\n\n Structured data is what you\u2019ll find in a table where each row is an observation, and each column is a variable or field. By contrast, unstructured data usually needs to be pre-processed before you can work with it in a data science project, or it can be used by specialist models that can process it internally. Examples of unstructured data include text, images, and sound. <\/p>\n\n\n\n As you might have guessed, unstructured data is used more in advanced and specialized subfields in data science, like natural language processing and computer vision. Most data scientists start with, and continue working with, structured data for many of their projects. I recommend that this is where you start, too.<\/p>\n\n\n\n I recommend you keep the notion of structured and unstructured data in mind as we explore standard data formats.<\/p>\n\n\n\n In addition to the quality of the data, we also have to choose between available data formats. You\u2019ll come across two broad types of data formats as a data scientist: downloadable data (often CSV) and databases. <\/p>\n\n\n\n Downloadable<\/em> data is nearly always structured data and often takes the form of comma-separated value (CSV) files. These downloads are available from various online repositories. They are among some of the most prolific and most accessible sources of data. If you\u2019re new to data exploration, this is the best place to get started, as they\u2019re easy to find, human-readable, and easy to work with without any extra steps. <\/p>\n\n\n\n If you\u2019re ready to enter the world of databases, it\u2019s worth understanding that they are further subdivided into relational (SQL) and non-relational (non-SQL) databases. As a broad rule, relational databases contain structured data and non-relational databases contain non-structured data, but determining whether data is structured<\/em> is not an exact science. Instead, think of non-relational databases as being adaptable to the shape of the data they are storing. <\/p>\n\n\n\n Databases are commonly used in the following cases: when you have large datasets, when multiple people need to access and modify the data simultaneously, when datasets need to be able to scale, and when data is unstructured (non-SQL only). In addition, if you\u2019re commissioned to do data analysis for your company, you may find that you\u2019re given a database to work with as it\u2019s already in-house. <\/p>\n\n\n\n PyCharm Professional has excellent support for SQL and non-SQL databases. If your work involves using various databases and writing SQL queries, you can check out our webinar on Visual SQL Development with PyCharm<\/em><\/a> to get more information about the functionality. Alternatively, you can learn how to explore tables without writing a single line of SQL<\/a> with PyCharm and import your dataset into PyCharm and explore it<\/a>. <\/p>\n\n\n\n \n \n Try PyCharm Professional for free\n <\/a>\n<\/p>\n\n\n\n Once you\u2019re ready to find out how to get data, there are plenty of resources you can download to use for your data science project. This is not an endless list, but it\u2019s a good place to start and a natural progression for your data science journey.<\/p>\n\n\n\n The UCI Machine Learning Repository<\/a> has over 600 datasets covering a host of exciting topics for you to explore, such as biology, health, physics, and climate. UCI datasets also have a diverse set of data types, including images, sequential, and time series. I recommend looking at a few different datasets and types of data if you\u2019re new to data science, as it will help you expand your understanding of what data often looks like. <\/p>\n\n\n\n Another well-known website for datasets is Kaggle<\/a>. Not only can you sign up to Kaggle to download datasets for data science projects, but it also has a large community of like-minded people who run company-sponsored competitions designed to help you develop your data science skills. If you\u2019re looking for a famous dataset<\/a> that you\u2019ve seen used in numerous examples, you\u2019ll almost certainly find it hosted on Kaggle.<\/p>\n\n\n\n Hugging Face<\/a> is another resource that is rich in datasets. You can filter the results by modalities, including audio, geospatial, and video, and provide a range for the size of your dataset, which can be particularly helpful when you want to start small. Hugging Face has many natural language and computer vision datasets, so you might want to head over there once you\u2019re past the basics and interested in more specialized fields.<\/p>\n\n\n\n There are many more places that you can go on your data science journey to find fun datasets to explore. You can check out GitHub<\/a> for curated open source datasets, FiveThirtyEight<\/a> for datasets relating to American politics and sports, and lastly, one of my favorites, the UK government<\/a>, to get datasets relating to public services and the economy in the UK. <\/p>\n\n\n\n Congratulations! You\u2019ve gained a better understanding of what \u201cgood data\u201d is, and you know where to look to find datasets for data science projects. Once you\u2019ve chosen a dataset, you\u2019re ready to start preparing and analyzing your data<\/a>. <\/p>\n\n\n\n Remember, you can use Jupyter notebooks inside PyCharm to explore both file format and database datasets<\/a>. <\/p>\n\n\n\n You can read<\/a> or watch a video<\/a> showing just some of the ways you can use Jupyter notebooks inside PyCharm to boost your productivity on your data science journey with your chosen dataset. <\/p>\n\n\n\nDo you want structured or unstructured data?<\/h2>\n\n\n\n
What are standard data formats?<\/h2>\n\n\n\n
Where can I find datasets for my data science projects?<\/h2>\n\n\n\n
UCI Machine Learning Repository<\/h3>\n\n\n\n
Kaggle<\/h3>\n\n\n\n
Hugging Face<\/h3>\n\n\n\n
Many more<\/h3>\n\n\n\n
What are the next steps?<\/h2>\n\n\n\n