🕷️ Scraping NAFDAC Greenbook from A to Z: How I Extracted 8,600+ Official Records into CSV

Author: Idris Abdulhamood | Category: Web Scraping, Data Engineering & Automation

The NAFDAC Greenbook (greenbook.nafdac.gov.ng) is the official public portal for all approved drugs, medical devices, herbals, and biologics registered by the Nigerian government.

For data analysts, researchers, or software engineers, having this database in a clean, structured .csv format opens up endless possibilities—from fraud detection systems to market research.

However, getting this data off the website and into an Excel sheet wasn't as simple as running pandas.read_html(). Here is the complete journey of how I scraped the entire portal from A to Z—including the wall I hit, why fast scripts failed, and the solution that successfully captured all 8,608 official entries.

🛑 Phase 1: The Trap – Why Standard Scraping Failed

My first approach was the standard Python scraping toolkit: requests and BeautifulSoup.

The strategy seemed straightforward: iterate through pages 1 to 901, download the HTML table rows, parse <td> elements, and output a DataFrame:

import requests
from bs4 import BeautifulSoup

# The naive approach
res = requests.get("https://greenbook.nafdac.gov.ng/?page=1")
soup = BeautifulSoup(res.text, 'html.parser')
table = soup.find('table')
print(table)  # Result: Found table header, but ZERO table rows!

The Output: ✅ SUCCESS! Downloaded and saved 0 official NAFDAC entries to 'nafdac_registered_drugs.csv'

💡 The Root Cause

The NAFDAC Greenbook uses client-side JavaScript (DataTables). When a web scraper sends a standard HTTP request, the server returns an empty HTML shell. The browser normally runs JavaScript after load to pull data via AJAX requests and render the rows dynamically. Since standard Python requests cannot execute JavaScript, it only saw an empty table frame!

⚡ Phase 2: The Need for Speed (And Why Multithreading Still Failed)

I tried boosting speed using concurrent.futures.ThreadPoolExecutor with 20 parallel workers across 901 pages.

 Progress: Downloaded 50 / 901 pages | Captured 0 entries...
 Progress: Downloaded 100 / 901 pages | Captured 0 entries...
 ...
 Progress: Downloaded 901 / 901 pages | Captured 0 entries...

While the script ran all 901 requests in 30 seconds, it still captured 0 entries. Speed doesn't matter if your HTTP client lacks a JavaScript execution engine!

🛠️ Phase 3: The Breakthrough – Headless Browser Automation

To get the actual rendered data, I switched to a Headless Chromium Browser using Selenium. A headless browser spins up a real Chrome instance in memory, executes all embedded JavaScript, populates the DataTables DOM, and allows full table inspection.

💡 Optimization Trick: Dropdown Batching

Instead of clicking through 901 separate pages at 10 rows per page, I configured Selenium to find the DataTables length dropdown (<select name="entries">) and change the display value to 100 entries per page. This reduced total page interactions from 901 down to ~90!

💻 The Working Production Code

Here is the complete Python script used to scrape the entire database:

# =====================================================================
# NAFDAC GREENBOOK HEADLESS SCRAPER ENGINE
# =====================================================================
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select

# 1. Initialize Headless Chrome Environment
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome(options=chrome_options)

url = "https://greenbook.nafdac.gov.ng/"
driver.get(url)
time.sleep(5)  # Allow DOM & DataTables JS to load completely

# 2. Expand Table Pagination to 100 Rows Per Page
try:
    select_elem = driver.find_element(By.NAME, "entries")
    select = Select(select_elem)
    select.select_by_value("100")
    time.sleep(3)
except Exception:
    pass

all_records = []
page = 1

# 3. Extract Rendered DOM Rows Page-by-Page
while True:
    rows = driver.find_elements(By.XPATH, "//table/tbody/tr")
    
    for row in rows:
        cols = [col.text.strip() for col in row.find_elements(By.TAG_NAME, "td")]
        if len(cols) >= 4 and cols[0] != "No matching records found":
            all_records.append(cols)

    print(f"Page {page} processed | Captured {len(all_records)} total entries...", flush=True)

    # Click 'Next' button until pagination ends
    try:
        next_btn = driver.find_element(By.XPATH, "//a[contains(text(), 'Next') or contains(@class, 'next')]")
        if "disabled" in next_btn.get_attribute("class"):
            break
        next_btn.click()
        time.sleep(1.2)
        page += 1
    except Exception:
        break

driver.quit()

# 4. Standardize Output Schema
standard_keys = [
    'Product Name', 'Active Ingredients', 'Category', 'NRN', 
    'Form', 'ROA', 'Strengths', 'Applicant Name', 'Approval Date', 'Status'
]

cleaned_rows = []
for item in all_records:
    row_dict = {standard_keys[i]: item[i] for i in range(min(len(item), len(standard_keys)))}
    cleaned_rows.append(row_dict)

# 5. Clean Data & Export CSV
df_full = pd.DataFrame(cleaned_rows)
df_full = df_full.dropna(subset=['NRN']).drop_duplicates(subset=['NRN'])
df_full.to_csv("nafdac_registered_drugs.csv", index=False)

print(f"\n✅ SUCCESS! Rendered and saved {len(df_full)} live entries to 'nafdac_registered_drugs.csv'")

📊 Phase 4: Final CSV Audit & Data Structure

The output file nafdac_registered_drugs.csv contained 8,608 unique records, fully cleaned with 0 duplicate Registration Numbers (NRNs).

Product Category	Total Records Captured
Herbals and Nutraceuticals	1,739
Veterinary Products	1,729
Medical Devices	1,727
Drugs & Pharmaceuticals	1,718
Vaccines and Biologics	1,695
TOTAL DATASET SIZE	8,608

💡 Key Takeaways for Web Scraping Dynamic Websites

Check for JavaScript Rendering First: Always inspect the page source vs rendered DOM elements before picking your Python scraping stack.
HTTP Request Libraries Have Limits: When DataTables or JS frameworks populate tables dynamically, tools like Selenium or Playwright are mandatory.
Optimize Display Limits: Check if the target portal allows displaying 50 or 100 entries per page. This significantly reduces total network calls and pagination clicks.
Deduplicate at the Source: Dropping duplicates based on unique primary keys (like NRNs) ensures high data hygiene right out of the box.

Search This Blog

ABDULHAMEED IDRIS ADEDAMOLA