Awesome

Notice

To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink. We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo. The last version before this decision is v0.64.1

We thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily

pipelinewise-tap-twilio

Singer tap that extracts data from a Twilio API and produces JSON-formatted data following the Singer spec.

This is a PipelineWise compatible tap connector.

This tap:

• Extracts the following resources:

  • accounts
  • addresses
  • dependent_phone_numbers
  • applications
  • available_phone_number_countries
  • available_phone_numbers_local
  • available_phone_numbers_mobile
  • available_phone_numbers_toll_free
  • incoming_phone_numbers
  • keys
  • calls
  • conferences
  • conference_participants
  • outgoing_caller_ids
  • recordings
  • usage_records
  • usage_triggers
  • transcriptions
  • queues
  • message_media
  • alerts

• Extracts TaskRouter resources:

  • workspaces
  • activities
  • events
  • tasks
  • task_channels
  • task_queues
  • cumulative_statistics
  • workers
  • worker_channels
  • workflows

• Extracts Programmable Chat resources (members and chat_messages are FULL_TABLE synced, so take care syncing them, they result in a lot of request/data):

  • services
  • roles
  • chat_channels
  • members
  • chat_messages
  • users

• Outputs the schema for each resource

• Incrementally pulls data based on the input state

Streams

Standard Endpoints:

accounts

• Endpoint: https://api.twilio.com/2010-04-01/Accounts.json
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

addresses

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses.json
• Parent: account
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

dependent_phone_numbers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses/{ParentId}/DependentPhoneNumbers.json
• Parent: addresses
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

applications

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Applications.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

available_phone_number_countries

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers.json
• Parent: accounts
• Primary key fields: country_code
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_local

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Local.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_mobile

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Mobile.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_toll_free

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/TollFree.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: none

incoming_phone_numbers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/IncomingPhoneNumbers.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

keys

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Keys.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

calls

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Calls.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

conferences

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

conference_participants

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences/{ParentId}/Participants.json
• Parent: conferences
• Primary key fields: uri
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

outgoing_caller_ids

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/OutgoingCallerIds.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

recordings

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Recordings.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

transcriptions

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Transcriptions.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

queues

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Queues.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

message_media

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{ParentId}/Media.json
• Parent: messages
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

usage_records

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Records.json
• Parent: accounts
• Primary key fields: account_sid, category, start_date
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

usage_triggers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Triggers.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

alerts

• Endpoint: https://monitor.twilio.com/v1/Alerts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

TaskRouter Endpoints:

workspaces

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

activities

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Activities
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

events

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Events
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

tasks

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Tasks
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

task_channels

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskChannels
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

task_queues

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

cumulative_statistics

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues/{ParentId}/CumulativeStatistics
• Parent: task_queues
• Replication strategy: FULL_TABLE
• Transformations: none

workers

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

channels

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers/{ParentId}/Channels
• Parent: workers
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

workflows

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workflows
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

Programmable Chat Endpoints:

services

• Endpoint: https://chat.twilio.com/v2/Services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

roles

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Roles
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

chat_channels

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

members

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Members
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

chat_messages

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Messages
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

users

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Users
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

Authentication

This tap authenticates to the Twilio API using Basic Auth.

To set up authentication simply include your Twilio account_sid and auth_token in the tap config.

Quick Start

1. Install

     make venv
   

2. Create your tap's config.json file. The api_key is available in the twilio Console UI (see Authentication above). The date_window_days is the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 30). The start_date is the absolute beginning date from which incremental loading on the initial load will start.

       {
           "account_sid": "YOUR_ACCOUNT_SID",
           "auth_token": "YOUR_AUTH_TOKEN",
           "start_date": "2019-01-01T00:00:00Z",
           "user_agent": "tap-twilio <api_user_email@your_company.com>",
       }
   

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "registers",
       "bookmarks": {
           "acounts": "2020-03-23T10:31:14.000000Z",
           "...": "2020-03-23T00:00:00.000000Z"
       }
   }
   

3. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-twilio --config config.json --discover > catalog.json
   

   See the Singer docs on discovery mode here.

4. Run the Tap in Sync Mode (with catalog)

   For Sync mode:

   > tap-twilio --config tap_config.json --catalog catalog.json
   

   Messages are written to standard output following the Singer specification. The resultant stream of JSON data can be consumed by a Singer target.

To run tests

Install python test dependencies in a virtual env and run tests

make venv test

To lint the code

Install python test dependencies in a virtual env and run linter

make venv pylint

Licence

GNU AFFERO GENERAL PUBLIC LICENSE