Extract SQL tables, insert, update, and delete rows in SQL databases through SQLAlchemy — In a data science project, we often need to interact with Relational databases, such as, extracting tables, inserting, updating, and deleting rows in SQL tables. To accomplish these tasks, Python has one such library, called SQLAlchemy. It supports popular SQL databases, such as PostgreSQL, MySQL, SQLite, Oracle, Microsoft SQL Server…

Extract Data from PDF Files Effectively — Background In the previous article, I talked about how to use tabula-py and Pandas in Python to scrape data from both structured and unstructured data from PDF files. In this article, I’m going to introduce an alternative way to scrape data from PDF files: PDFQuery. Required Libraries PDFQuery: to scrape text from PDF…

The differences in terms of cost functions, Ordinary Least Squares (OLS), Gradient Descent (GD), and Maximum Likelihood Estimation (MLE). — In this article, I’ll cover a roadmap for statistical model development. Specifically, we’ll talk about how the development of linear regression differs from logistic regression. We’ll also discuss why some methods work in one model, but not in another model. The following is an overview of this roadmap.

Observational Study, Experimental Study, Regression Model, Instrumental Variable, Difference-in-difference Model, Parametric Test and Non-Parametric Test — As data science practitioners, we’ve been asked a lot that “Does X drive Y?” Y is the outcome that we care about. X could be a new feature, product or drug. For example, a website owner would like to ask if a new web page design leads to a higher…

Understand OLS Linear Regression with a bit of math — The OLS estimator is known to be unbiased, consistent and BLUE (Best Linear Unbiased Estimator). But what do these properties mean? Why are they important for a linear regression model? In this article, we will discuss these properties. A typical linear regression looks like something as follows. The response variable…

Discussion of Exogenous variable, Exogenous variable, Omitted variable, Measurement Error, and Simultaneity Bias — In my previous article, we discussed some common issues when designing a linear regression — Omitting Important Variables and Including Irrelevant Variables. In this article, we’ll discuss Endogeneity in a linear regression model, especially in the context of Causal Inference. A linear regression model is a popular tool used to…

Geocoding, Reverse Geocoding, Compute Surface Distance, Driving Distance, Create Scatter Map, Bubble Map, Choropleth Map and Animated Map in Python — Geospatial data science is booming now. A typical Geospatial dataset includes data, such as, latitude, longitude, addresses, street names, zip codes. These data require a different set of tools to process and turn them into insights. In this article, I’ll share Python libraries that are useful to process geospatial data…

An efficient way to OCR scanned images — Oftentimes, you would like to make scanned files or image files searchable in PDF, because it is much quicker and more convenient to search keywords with a searchable PDF than in native image format. More importantly, the underlying text would be useful for Natural Language Processing (NLP). In this article…

Host a Web App Publicly with Github and Heroku — Data scientists and machine learning engineers spend a lot of their time on Exploratory Data Analysis (EDA), ML model development, and Hyper -Parameter Tuning. Often time, the ultimate goal of a project is to deploy the machine learning model in production. …