Home

Awesome

Python support for IBM Db2 for LUW and IBM Db2 for z/OS

Python, DB-API components for IBM Db2 for LUW and IBM Db2 for z/OS

Provides Python interface for connecting to IBM Db2 for LUW and IBM Db2 for z/OS

<a name='components'></a>

Components

  1. The ibm_db contains:
    • ibm_db driver: Python driver for IBM Db2 for LUW and IBM Db2 for z/OS databases. Uses the IBM Data Server Driver for ODBC and CLI APIs to connect to IBM Db2 for LUW.
    • ibm_db_dbi: Python driver for IBM Db2 for LUW that complies to the DB-API 2.0 specification.

<a name="api"></a> API Documentation

For more information on the APIs supported by ibm_db, please refer to below link:

https://github.com/ibmdb/python-ibmdb/wiki/APIs

<a name="prereq"></a>

Pre-requisites

Install Python 3.7 <= 3.12. The minimum python version supported by driver is python 3.7 and the latest version supported is python 3.12.

To install ibm_db on z/OS system

Please follow detailed installation instructions as documented here: ibm_db Installation on z/OS

For MacOS M1/M2/ Apple Silicon chip system

Important: The driver for IBM DB2 is not compatible with Apple Silicon and will have to run in emulated mode. Please support this request for an Apple Silicon version of the driver to show IBM that you are interested in a native solution.

Several things might be necessary to get ibm_db working on the Apple Silicon architecture:

Linux/Unix:

If you face problems due to missing python header files while installing the driver, you would need to install python developer package and retry install. e.g:

    zypper install python-devel
     or
    yum install python-devel

Windows:

MAC OS:

Db2 V11.5.4 clidriver is built with GCC version 8.4.0 and hence you may need to upgrade to this version if you face problems due to old version of the GCC compiler in your environment while loading the ibm_db library.

Docker Linux containers:

Manual Installation

To install ibm_db from source code after clone from git, or to know the detialed installation steps for various platforms, please check INSTALL document.

<a name="installation"></a>

Installation

You can install the driver using pip as:

pip install ibm_db

This will install ibm_db and ibm_db_dbi module.

Note: When we install ibm_db package on Linux, MacOS and Windows, pip install ibm_db command install prebuilt Wheel package that includes clidriver too and ignores IBM_DB_HOME or IBM_DB_INSTALLER_URL environment variables if set. Also, auto downloading of clidriver does not happen as clidriver is already present inside wheel package.

To inforce auto downloading of clidriver or to make setting of environment variable IBM_DB_HOME effective, install ibm_db from source distribution using below command:

pip install ibm_db --no-binary :all: --no-cache-dir

If you have to use your own URL for clidriver.tar.gz/.zip please set environment variable

IBM_DB_INSTALLER_URL=full_path_of_clidriver.tar.gz/.zip

When ibm_db get installed from wheel package, you can find clidriver under site_packages directory of Python. You need to copy license file under site_packages/clidriver/license to be effective, if any.

Note: For windows after installing ibm_db, recieves the below error when we try to import ibm_db :

Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ibm_db
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing ibm_db: The specified module could not be found.
>>>

We need to make sure to set dll path of dependent library of clidriver before importing the module as:

import os
os.add_dll_directory('path to clidriver installation until bin')
import ibm_db

e.g:
os.add_dll_directory('C:\\Program Files\\IBM\\CLIDRIVER\\bin')
import ibm_db

Refer https://bugs.python.org/issue36085 for more details.

yum install python gcc pam wget python-devel.x86_64 libxml2
use, `yum install python3` to install python 3.x

if pip or pip3 does not exist, install it as:
wget https://bootstrap.pypa.io/get-pip.py
docker cp get-pip.py /root:<containerid>
cd root
python2 get-pip.py or python3 get-pip.py

Install python ibm_db as:
pip install ibm_db
or
pip3 install ibm_db


pip uninstall ibm_db

The ODBC and CLI Driver(clidriver) is automatically downloaded at the time of installation and it is recommended to use this driver. However, if you wish to use an existing installation of clidriver or install the clidriver manually and use it, you can set IBM_DB_HOME environment variable as documented below:

Note: You must need to install ibm_db using command pip install ibm_db --no-binary :all: --no-cache-dir on Linux, Windows and MacOS to make setting of IBM_DB_HOME effective.

You are required to set the library path to the clidriver under IBM_DB_HOME to pick this version of the ODBC and CLI Driver.<br> e.g:

Windows:
set LIB=%IBM_DB_HOME%/lib;%LIB%

AIX:
export LIBPATH=$IBM_DB_HOME/lib:$LIBPATH

MAC:
export DYLD_LIBRARY_PATH=$IBM_DB_HOME/lib:$DYLD_LIBRARY_PATH

Other platforms:
export LD_LIBRARY_PATH=$IBM_DB_HOME/lib:$LD_LIBRARY_PATH

The ODBC and CLI driver is available for download at Db2 LUW ODBC and CLI Driver. Refer to (License requirements) for more details on the CLI driver for manual download and installation.

conda install -c conda-forge ibm_db
PlatformArchitectureSupportedVersion
Linuxamd64 (x86_64)YesLatest
Linuxppc64leYesLatest
DarwinMac OS x64YesLatest
Windowsx64YesLatest
Windowsx32YesLatest

Install Intel/x64 version of python on M1 Chip system. Check this link for detailed instructions.

<a name="quick example"></a> Quick Example

$ python
Python 3.6.5 (default, May 10 2018, 00:54:55)
[GCC 4.3.4 [gcc-4_3-branch revision 152973]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ibm_db
>>> #For connecting to local database named pydev for user db2inst1 and password secret, use below example
>>> #ibm_db_conn = ibm_db.connect('pydev', 'db2inst1', 'secret')
>>> #For connecting to remote database named pydev for uid db2inst and pwd secret on host host.test.com, use below example
>>> # Connect using ibm_db
>>> conn_str='database=pydev;hostname=host.test.com;port=portno;protocol=tcpip;uid=db2inst1;pwd=secret'
>>> ibm_db_conn = ibm_db.connect(conn_str,'','')
>>>
>>> # Connect using ibm_db_dbi
>>> import ibm_db_dbi
>>> conn = ibm_db_dbi.Connection(ibm_db_conn)
>>> # create table using ibm_db
>>> create="create table mytable(id int, name varchar(50))"
>>> ibm_db.exec_immediate(ibm_db_conn, create)
<ibm_db.IBM_DBStatement object at 0x7fcc5f44f650>
>>>
>>> # Execute tables API
>>> conn.tables('DB2INST1', '%')
[{'TABLE_CAT': None, 'TABLE_SCHEM': 'DB2INST1', 'TABLE_NAME': 'MYTABLE', 'TABLE_TYPE': 'TABLE', 'REMARKS': None}]
>>>
>>> # Insert 3 rows into the table
>>> insert = "insert into mytable values(?,?)"
>>> params=((1,'Sanders'),(2,'Pernal'),(3,'OBrien'))
>>> stmt_insert = ibm_db.prepare(ibm_db_conn, insert)
>>> ibm_db.execute_many(stmt_insert,params)
3
>>> # Fetch data using ibm_db_dbi
>>> select="select id, name from mytable"
>>> cur = conn.cursor()
>>> cur.execute(select)
True
>>> row=cur.fetchall()
>>> print("{} \t {} \t {}".format(row[0],row[1],row[2]),end="\n")
(1, 'Sanders')   (2, 'Pernal')   (3, 'OBrien')
>>> row=cur.fetchall()
>>> print(row)
[]
>>>
>>> # Fetch data using ibm_db
>>> stmt_select = ibm_db.exec_immediate(ibm_db_conn, select)
>>> cols = ibm_db.fetch_tuple( stmt_select )
>>> print("%s, %s" % (cols[0], cols[1]))
1, Sanders
>>> cols = ibm_db.fetch_tuple( stmt_select )
>>> print("%s, %s" % (cols[0], cols[1]))
2, Pernal
>>> cols = ibm_db.fetch_tuple( stmt_select )
>>> print("%s, %s" % (cols[0], cols[1]))
3, OBrien
>>> cols = ibm_db.fetch_tuple( stmt_select )
>>> print(cols)
False
>>>
>>> # Close connections
>>> cur.close()
True
>>> # Dropping the table created
>>> drop = "drop table mytable"
>>> stmt_delete = ibm_db.exec_immediate(ibm_db_conn,drop)
>>> conn1.tables('DB2INST1','MY%')
[]
>>>
>>> ibm_db.close(ibm_db_conn)
True

Logging

Logging in ibm_db Module

You can enable logging in the ibm_db module to debug and trace activities. Logging can be directed to the console or a specified file.

To enable logging:

import ibm_db

# Log to console
ibm_db.debug(True)

# Log to a file (e.g., log.txt)
ibm_db.debug("log.txt")

# stop logging
ibm_db.debug(False)

Calling ibm_db.debug(True) with boolean True argument will output logs to the console.

Calling ibm_db.debug("log.txt") will log messages to the specified file (log.txt in this example).

Calling ibm_db.debug(False) with boolean False argument will stop logging.

Logging in ibm_db_dbi Module

You can enable logging in the ibm_db_dbi module to debug and trace activities. Logging can be directed to the console or a specified file.

To enable logging:

import ibm_db_dbi as dbi

# Log to console
dbi.debug(True)

# Log to a file (e.g., log.txt)
dbi.debug("log.txt")

# stop logging
dbi.debug(False)

Calling dbi.debug(True) with boolean True argument will output logs to the console.

Calling dbi.debug("log.txt") will log messages to the specified file (log.txt in this example).

Calling dbi.debug(False) with boolean False argument will stop logging.

Example of SSL Connection String

Using SSLServerCertificate keyword
 
connStr = "DATABASE=<DATABASE_NAME>;HOSTNAME=<HOSTNAME>;PORT=<SSL_PORT>;UID=<USER_ID>;PWD=<PASSWORD>;" +
          "SECURITY=SSL;SSLServerCertificate=<FULL_PATH_TO_SERVER_CERTIFICATE>;"
conn = ibm_db.connect(connStr,'','')

Note the two extra keywords Security and SSLServerCertificate used in connection string. SSLServerCertificate should point to the SSL Certificate from server or an CA signed certificate. Also, PORT must be SSL port and not the TCPI/IP port. Make sure Db2 server is configured to accept connection on SSL port else ibm_db will throw SQL30081N error.

Value of SSLServerCertificate keyword must be full path of a certificate file generated for client authentication. It normally has *.arm or *.cert or *.pem extension. ibm_db do not support *.jks format file as it is not a certificate file but a Java KeyStore file, extract certificate from it using keytool and then use the *.cert file.

ibm_db uses IBM ODBC/CLI Driver for connectivity and it do not support a *.jks file as keystoredb as keystore.jks is meant for Java applications. Note that *.jks file is a Java Key Store file and it is not an SSL Certificate file. You can extract SSL certificate from JKS file using below keytool command:

keytool -exportcert -alias your_certificate_alias -file client_cert.cert -keystore  keystore.jks

Now, you can use the generated client_cert.cert as the value of SSLServerCertificate in connection string.

Do not use keyworkds like sslConnection=true in connection string as it is a JDBC connection keyword and ibm_db ignores it. Corresponding ibm_db connection keyword for sslConnection is Security hence, use Security=SSL; in connection string instead.

ibm_db supports only ODBC/CLI Driver keywords in connection string: https://www.ibm.com/docs/en/db2/11.5?topic=odbc-cliodbc-configuration-keywords

connStr = "DATABASE=database;HOSTNAME=hostname;PORT=port;PROTOCOL=TCPIP;UID=username;PWD=passwd;Security=SSL";

We just need to add Security=SSL in connection string to have a secure connection against Db2 server in IBM Cloud.

Note: You can also create a KeyStore DB using GSKit command line tool and use it in connection string along with other keywords as documented in DB2 Infocenter.

Using SSLClientKeyStoreDB and SSLClientKeyStoreDBPassword keyword
 
connStr = "DATABASE=<DATABASE_NAME>;HOSTNAME=<HOSTNAME>;PORT=<SSL_PORT>;UID=<USER_ID>;PWD=<PASSWORD>;" +
          "SECURITY=SSL;SSLClientKeyStoreDB=<FULL_PATH_TO_KEY_STORE_DB>;SSLClientKeyStoreDBPassword=<KEYSTORE_PASSWD>;"
conn = ibm_db.connect(connStr,'','')

Using SSLClientKeyStoreDB and SSLClientKeyStash keyword
 
connStr = "DATABASE=<DATABASE_NAME>;HOSTNAME=<HOSTNAME>;PORT=<SSL_PORT>;UID=<USER_ID>;PWD=<PASSWORD>;" +
          "SECURITY=SSL;SSLClientKeyStoreDB=<FULL_PATH_TO_KEY_STORE_DB>;" +
          "SSLClientKeyStash=<FULL_PATH_TO_CLIENT_KEY_STASH>;"
conn = ibm_db.connect(connStr,'','')

If you have downloaded IBMCertTrustStore from IBM site, ibm_db will not work with it; you need to download Secure Connection Certificates.zip file that comes for IBM DB2 Command line tool(CLP). Secure Connection Certificates.zip has *.kdb and *.sth files that should be used as the value of SSLClientKeystoreDB and SSLClientKeystash in connection string.

<a name="Licenserequirements"></a>For z/OS and iSeries Connectivity and SQL1598N error

Troubleshooting SQL1598N Error:

If you have copied db2con*.lic file under clidriver/license directory, but still getting SQL1598N Error; try below options:

  db2cli validate -connstring "connection string as used in python program" -displaylic

  OR

  db2cli validate -database "dbname:hostname:port" -userid dbuser -passwd dbpasswd -connect -displaylic

If you intend to install the clidriver manually, Following are the details of the client driver versions that you can download from CLIDRIVER to be able to connect to databases on non-LUW servers. You would need the client side license file as per Version for corresponding installation.:

<a name="LicenseDetails"></a> CLIDriver and Client license versions for Specific Platform and Architecture

PlatformArchitectureCli DriverSupportedVersion
AIXppcaix32_odbc_cli.tar.gzYesV11.5.8
othersaix64_odbc_cli.tar.gzYesV11.5.8
Darwinx64macos64_odbc_cli.tar.gzYesV11.5.8
Linuxx64linuxx64_odbc_cli.tar.gzYesV11.5.8
s390xs390x64_odbc_cli.tar.gzYesV11.5.8
s390s390_odbc_cli.tar.gzYesV11.1
ppc64 (LE)ppc64le_odbc_cli.tar.gzYesV11.5.8
ppc64ppc64_odbc_cli.tar.gzYesV10.5
ppc32ppc32_odbc_cli.tar.gzYesV10.5
otherslinuxia32_odbc_cli.tar.gzYesV11.5.8
Windowsx64ntx64_odbc_cli.zipYesV11.5.8
x32nt32_odbc_cli.zipYesV11.5.8
Suni86pcsunamd64_odbc_cli.tar.gzYesV10.5
sunamd32_odbc_cli.tar.gzYesV10.5
sparcsun64_odbc_cli.tar.gzYesV11.1
sparcsun32_odbc_cli.tar.gzYesV11.1

You can refer to ODBC and CLI Driver installation for details on how to install the driver manually.

<a name='downloads'></a>

Downloads

Use following pypi web location for downloading source code and binaries ibm_db: https://pypi.python.org/pypi/ibm_db . You can also get the source code by cloning the ibm_db github repository as :

git clone git@github.com:ibmdb/python-ibmdb.git

<a name='support'></a>

Support & feedback

Your feedback is very much appreciated and expected through project ibm-db:

<a name='contributing-to-the-ibm_db-python-project'></a>

Contributing to the ibm_db python project

See CONTRIBUTING

The developer sign-off should include the reference to the DCO in remarks(example below):
DCO 1.1 Signed-off-by: Random J Developer <random@developer.org>

<a name='KnownIssues'></a>

Some common issues

1. Installation Issues for missing python.h file

Always use the latest pip

python3 -m pip install --upgrade pip

Install the package python3-devel that delivers the python.h header file

For RHEL use
sudo yum install python3-devel

For Ubuntu use
sudo apt-get install python3-devel

2. SQL30081N Error

If connection fails with SQL30081N error - means ibm_db installation is correct and there is some issue with connection string. Please check database connection info and use correct connection string. If you are using SSL connection, port must be SSL port and connection string must have Security=SSL; and SSLServerCertificate=<full path of cert.arm file>;.

3. Issues with MAC OS X

>>> import ibm_db
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/usr/local/lib/python3.5/site-packages/ibm_db.cpython-35m-darwin.so, 2): Library not loaded: libdb2.dylib
  Referenced from: /usr/local/lib/python3.5/site-packages/ibm_db.cpython-35m-darwin.so
  Reason: image not found

You would need to set DYLD_LIBRARY_PATH to point to lib folder as per the installation location of clidriver in your environment. Assuming the driver is installed at /usr/local/lib/python3.5/site-packages/clidriver, you can set the path as:

export DYLD_LIBRARY_PATH=/usr/local/lib/python3.5/site-packages/clidriver/lib:$DYLD_LIBRARY_PATH

If the issue is not resolved even after setting DYLD_LIBRARY_PATH, you could refer: MAC OS Hints and Tips

Resolving SQL1042C error

If you hit following error while attempting to connect to a database:

>>> import ibm_db
>>> ibm_db.connect("my_connection_string", "", "")
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 Exception: [IBM][CLI Driver] SQL1042C An unexpected system error occurred. SQLSTATE=58004 SQLCODE=-1042

Set DYLD_LIBRARY_PATH to point to icc folder as per the installation location of clidriver in your environment.

export DYLD_LIBRARY_PATH=/usr/local/lib/python3.5/site-packages/clidriver/lib/icc:$DYLD_LIBRARY_PATH

In case of similar issue in windows platform

set PATH=<clidriver_folder_path>\bin\amd64.VC12.CRT;%PATH%

4. ERROR: Failed building wheel for ibm_db

In case of the error seen while building wheel use the following flag along with ibm_db for installation

To install the package ibm_db it is necessary at first to install the build dependency package - wheel:
pip3 install wheel
Install ibm_db with the pip flag --no-build-isolation:
pip3 install ibm_db --no-build-isolation

5. For Issues on IBM iSeries System (AS400)

<a name='testing'></a>

Testing

Tests displaying Python ibm_db driver code examples are located in the ibm_db_tests directory. A valid config.py will need to be created to configure your Db2 settings. A config.py.sample exists that can be copied and modified for your environment.

For example, by sourcing the following ENV variables:
For Linux
export DB2_USER=<Username>
export DB2_PASSWD=<Password>

For windows
set DB2_USER=<Username>
set DB2_PASSWD=<Password>


If not using environment variables, update user and password information in
config.json file.

The config.py should look like this:

test_dir =      'ibm_db_tests'         # Location of testsuite file (relative to current directory)
file_path = 'config.json'

with open(file_path, 'r') as file:
    data = json.load(file)

database = data['database']               # Database to connect to
hostname = data['hostname']               # Hostname
port = data['port']                       # Port Number

env_not_set = False
if 'DB2_USER' in os.environ:
     user = os.getenv('DB2_USER')         # User ID to connect with
else:
    user = data['user']
    env_not_set = True
if 'DB2_PASSWD' in os.environ:
    password = os.getenv('DB2_PASSWD')    # Password for given User ID
else:
    password = data['password']
    env_not_set = True

if env_not_set == True:
    warnings.warn("Warning: Environment variable DB2_USER or DB2_PASSWD is not set.")
    print("Please set it before running test file and avoid")
    print("hardcoded password in config.json file." )

Point the database to mydatabase as created by the following command.

The tests that ibm_db driver uses depends on a UTF-8 database. This can be created by running:

  CREATE DATABASE mydatabase USING CODESET UTF-8 TERRITORY US

Some of the tests utilize XML functionality only available in version 9 or later of Db2. While Db2 v8.x is fully supported, two of the tests (test_195.py and test_52949.py) utilize XML functionality. These tests will fail on version 8.x of Db2.

Running the driver testsuite on Linux

In order to run the entire python driver testsuite on Linux, run this command at the command prompt:

  python ibmdb_tests.py

To run a single test, set the environment variable, SINGLE_PYTHON_TEST, to the test filename you would like to run, followed by the previous command.

Running the driver testsuite on Windows

In order to run the entire python driver testsuite on Windows, run this command at the command prompt:

  python ibmdb_tests.py

To run a single test, set the environment variable, SINGLE_PYTHON_TEST, to the test filename you would like to run, followed by the previous command.

Known Limitations for the Python driver

If trusted context is not set up, there will be two failures related to trusted context. When thick client has been used then additional three failures related to create, recreate DB.

ibm_db do not work with arm64 version of python, use x64 version.

Known Limitations for the Python wrapper

  1. The rowcount for select statements can not be generated.
  2. Some warnings from the drivers are not caught by the wrapper. As such these might go unnoticed.

M1 MAC Steps to Install IBM DB and Support Docker RUN

Installation Steps:

Install Podman / Docker:

Install Intel Version of Python:

  1. Download and install the Intel version of Python 3.8 from the official Python website.
  2. After installation, ensure that you set the interpreter to Python 3.8 in your development environment.

Configuration Steps:

Configure Docker for M1:

Build Docker Image (Intel):

Additional Notes: