READING Connect to Fminer Database Using Python
On This Page

    Connect to Fminer Database Using Python

    Connect to Fminer Database Using Python

    Introduction

    Fminer is a powerful web scraping tool as well as the best browser automation tool that supports many features that web scraping software needs. Some of the Fminer’s key features are Support of Multithreading, Captcha solving feature, setting actions to deal with browser automation like Input text into the text field, selecting options from the drop-down, choosing radio buttons and checkboxes, project scheduling and email feature, custom  Proxy support and many more. You can read details on each feature of Fminer in my previous post Fminer – Visual Web Scraping Software With Macro Recorder and Diagram Designer.

    In this post, I am going to shed some light on one of the compelling features of Fminer which is called “Run Code action”. Using this feature, one can run custom Python code to manipulate scraped data, to upload scraped data directly into MySQL or SQL server from Fminer, and you can do any tricky thing using Python code you can write.

    Most web scrapers do not have this feature using which you can use to put your code in the scraping workflow for data manipulation, to control the scraping process, or for any other tricky work. Some of the data extraction software that has this feature are Visual Web Ripper and Fminer. I am sharing Python code you can use inside Fminer and connect it with MySQL or SQL Server.

    Python code to read data from the MySQL database and insert it into the Fminer table

    
    Fminer Run Code
    
    
    
    
    
    
    
    
    
    
    
    
    
    import sys
    import pyodbc
    table1=tables['student']
    connection_str ="""Driver={MySQL ODBC 5.3 ANSI Driver};Server=localhost;
    Database=student;User=root; Password=;Option=3;"""
    cnxn = pyodbc.connect(connection_str)
    cnxn.autocommit = True
    cursor=cnxn.cursor()
    sql="""select name from students"""
    cursor.execute(sql)
    rows=cursor.fetchall()
    for row in rows:
       	r=[{'name':row.name}]
    	   table1.add_rows(r)
    	   tables.update_ui()
    cursor.close()
    del cursor
    cnxn.close()
    
    

    import sys
    import pyodbc
    table1 = tables['keywords']
    connection_str ="""Driver={SQL Server Native Client 11.0};Server=SR1\SQLEXPRESS;
    Database=dbname;Trusted_Connection=yes;"""
    db_connection = pyodbc.connect(connection_str)
    db_connection.autocommit = True
    db_cursor = db_connection.cursor()
    sql_command="""select * from search_keywords"""
    db_cursor.execute(sql_command)
    rows = db_cursor.fetchall()
    for row in rows:
        	v=row.keyword
    	    row = [{'key': v}]        
    	    table1.add_rows(row)
    db_cursor.close()
    del db_cursor
    db_connection.close()
    

    Conclusion

    Both of the above codes read data from a database table in either MySQL or SQL server and insert it into the Fminer table and then Fminer can use it further based on the scraping workflow you designed.

    You can also use the table run code template and add database code into it with insert into query when you want to save scraped data back to database.  There are many  web scraping scenarios where you can utilize this code. For example you have list of keywords available in database and you want to search for that keyword on some website and then scrape data based on that keyword regularly and want to save scraped data back to the database so in that case you just need to schedule Fminer to scrape and all scraping and saving back to the database goes smooth way.

    Comments are closed.

    Keval Kothari

    Keval Kothari

    Founder and CEO