Awesome
Pandas.jl
This package provides a Julia interface to the excellent Pandas package from Python. It sticks closely to the Pandas API. One exception is that integer-based indexing is automatically converted from Python's 0-based indexing to Julia's 1-based indexing.
For a pure Julia alternative, please check out the excellent DataFrames.jl package.
Installation
Simply install the Pandas
package using the Julia package manager. From the Julia REPL:
using Pkg
Pkg.add("Pandas")
using Pandas
Which version of the Python Pandas library is used depends on how your installation of PyCall.jl is configured. By default, the Python Pandas library will be automatically downloaded and installed in a mininal Python installation managed by Julia and independent from any other Python distributions on your system.
See the PyCall configuration for instructions on changing this behavior.
Usage
In general, if df
is a Pandas object (such as a dataframe or series), then the Python command df.x(y, w=z)
becomes x(df, y, w=z)
in Julia. df.loc[a,b,c]
becomes loc(df)[a,b,c]
(same for iloc
and ix
). Example:
>> using Pandas
>> df = DataFrame(Dict(:age=>[27, 29, 27], :name=>["James", "Jill", "Jake"]))
age name
0 27 James
1 29 Jill
2 27 Jake
[3 rows x 2 columns]
>> describe(df)
age
count 3.000000
mean 27.666667
std 1.154701
min 27.000000
25% 27.000000
50% 27.000000
75% 28.000000
max 29.000000
[8 rows x 1 columns]
df[:age]
0 27
1 29
2 27
Name: age, dtype: int64
>> df2 = DataFrame(Dict(:income=>[45, 101, 87]), index=["Jake", "James", "Jill"])
>> df3 = merge(df, df2, left_on="name", right_index=true)
age name income
0 27 James 101
1 29 Jill 87
2 27 Jake 45
[3 rows x 3 columns]
>> iloc(df3)[1:2, 2:3]
name income
0 James 101
1 Jill 87
[2 rows x 2 columns]
>> mean(groupby(df3, "age")) #Or groupby(df, "age3") |> mean
income
age
27 73
29 87
[2 rows x 1 columns]
>> query(df3, :(income>85)) # or query(df3, "income>85")
age name income
0 27 James 101
1 29 Jill 87
[2 rows x 3 columns]
>> Array(df3)
3x3 Array{Any,2}:
27 "James" 101
29 "Jill" 87
27 "Jake" 45
>> plot(df3)
Input/Output
Example:
df = read_csv("my_csv_file.csv") # Read in a CSV file as a dataframe
to_json(df, "my_json_file.json") # Save a dataframe to disk in JSON format
Performance
Most Pandas operations on medium to large dataframes are very fast, since the overhead of calling into the Python API is small compared to the time spent inside Pandas' highly efficient C implementation.
Setting and getting individual elements of a dataframe or series is slow however, since it requires a round-trip of communication with Python for each operation. Instead, use the values
method to get a version of a series or homogeneous dataframe that requires no copying and is as fast to access and write to as a Julia native array. Example:
>> x_series = Series(randn(10000))
>> @time x_series[1]
elapsed time: 0.000121945 seconds (2644 bytes allocated)
>> x_values = values(x_series)
>> @time x_values[1]
elapsed time: 2.041e-6 seconds (64 bytes allocated)
>> x_native = randn(10000)
>> @time x_native[1]
elapsed time: 2.689e-6 seconds (64 bytes allocated)
Changes to the values(...) array propogate back to the underlying series/dataframe:
>> iloc(x_series)[1]
-0.38390854447454037
>> x_values[1] = 10
>> iloc(x_series)[1]
10
Caveats
Panels-related functions are still unwrapped, as well as a few other obscure functions. Note that even if a function is not wrapped explicitly, it can still be called using various methods from PyCall.