As of May 2023 release, the padding feature has some limitations:

• Padding adjustments are per side; no shortcut to  adjust top, bottom, left and right  at one go
• No way to define a default padding at report level

To overcome this issue, we can control the Power BI visual padding using themes.  We will use a simple theme that only adjust the padding for demonstration. Please save the below snippet as a json file so that we can use it as  a theme.

{
    "name": "Padding Demo",
    "visualStyles": {
        "*": {
            "*": {
                "padding":[{
                    "top":20,
                    "bottom":20,
                    "left": 20,
                    "right":20
                 }]
            }
        }

    }
}

The post Power BI Visual Padding using themes appeared first on Techno Whisp.

1. Installing Kaggle API

You can run pip install kaggle  to install the api. You might need to run pip install –user kaggle  on Linux or Mac if you are encountering issues with the installation.

2. Setting up the API Key

Kaggle API requires an API token. Go to the Account Tab (https://www.kaggle.com/<username>/account ) and click ‘Create API Token’.  A file named kaggle.json will be downloaded. Move this file in to ~/.kaggle/  folder in Mac and Linux or to C:\Users\<username>\.kaggle\  on windows. This is required for authentication and do not skip this step.

Alternatively, you can populate KAGGLE_USERNAME and KAGGLE_KEY  environment variables with values from kaggle.json to get the api to authenticate. Please note that environment variables have precedence over the kaggle.json file and hence setting them incorrectly will result in authentication failure even if you have correct contents in kaggle.json file.

3. Initializing and Authenticating

You can use below lines of code to get an authenticated API instance.

from kaggle.api.kaggle_api_extended import KaggleApi
api = KaggleApi()
api.authenticate()

4. Interacting with competitions

4.1 Searching competitions

# Searching competitions
# Signature: competitions_list(group=None, category=None, sort_by=None, page=1, search=None)
competitions = api.competitions_list(search='cat',category="playground")

# competitions is a list of competition objects.
# iterate though each item to access individual competition
for comp in competitions:
    print(comp.ref,comp.reward,comp.userRank,sep=',')

Most of the list methods in the api have a command line counter part which can be used to display the formatted results. It is not really useful for automation tasks as they dont return anything, but might be useful when exploring the api.

4.2 Listing and downloading competition Files

# List files for a competition
# Signature: competitions_data_list_files(id, **kwargs)
api.competitions_data_list_files('titanic')

# Download all files for a competition
# Signature: competition_download_files(competition, path=None, force=False, quiet=True)
api.competition_download_files('titanic')

# Download single file for a competition
# Signature: competition_download_file(competition, file_name, path=None, force=False, quiet=False)
api.competition_download_file('titanic','gender_submission.csv')

4.3 Submitting to competitions

# Signature: competition_submit(file_name, message, competition, quiet=False)
api.competition_submit('gender_submission.csv','API Submission','titanic')

4.4 Retrieving Leader Board

# Signature: competition_view_leaderboard(id, **kwargs)
leaderboard = api.competition_view_leaderboard('titanic')

5. Interacting with datasets

5.1 Searching datasets

# Signature: dataset_list(sort_by=None, size=None, file_type=None, license_name=None, tag_ids=None, search=None, user=None, mine=False, page=1, max_size=None, min_size=None) 
datasets=api.dataset_list(search='demographics',license_name='cc', file_type='csv')

# datasets is a collection of dataset
for dat in  datasets:
     print(dat.ref,dat.viewCount,dat.voteCount,sep=',')

5.2 Listing dataset files

#Signature: dataset_list_files(dataset)
# dataset string should be in format [owner]/[dataset-name]
api.dataset_list_files('avenn98/world-of-warcraft-demographics').files

5.3 Downloading Files

# Download all files of a dataset
# Signature: dataset_download_files(dataset, path=None, force=False, quiet=True, unzip=False)
api.dataset_download_files('avenn98/world-of-warcraft-demographics')

# download single file
#Signature: dataset_download_file(dataset, file_name, path=None, force=False, quiet=True)
api.dataset_download_file('avenn98/world-of-warcraft-demographics','WoW Demographics.csv')

6. Interacting with Kernels

6.1 Searching Kernels

#Signature: kernels_list(page=1, page_size=20, dataset=None, competition=None, parent_kernel=None, search=None, mine=False, user=None, language=None, kernel_type=None, output_type=None, sort_by=None)
kernels = api.kernels_list(search='titanic')
for kernel in kernels:
     print(kernel.ref,kernel.totalVotes,kernel.language,sep=',')

6.2 Retrieve a kernels output

# Retrieve output for a specified kernel
# Signature: kernels_output(kernel, path, force=False, quiet=True)
api.kernels_output('startupsci/titanic-data-science-solutions',path='.')

6.3 Get the status of the latest kernel run

# Signature: kernels_status(kernel)
api.kernels_status('startupsci/titanic-data-science-solutions')

6.4 Pull a kernel to local machine

# Signature: kernels_pull(kernel, path, metadata=False, quiet=True)
api.kernels_pull('startupsci/titanic-data-science-solutions',path='.')

6.5 Initialize metadata file for a kernel

# Signature: kernels_initialize(folder)
api.kernels_initialize('./demo')

6.6 Pushing a kernel to Kaggle

# Need to have a valid metadata file called 'kernel-metadata.json' in the folder
# Create one using kernels_initialize if you dont have one
# Signature: kernels_push(folder)
api.kernels_push('./demo')

Have any questions? Please add it as comments and I will try my best to answer them.

The post Kaggle API – The Missing Python Documentation appeared first on Techno Whisp.

1.  Downloading and Installing Anaconda

You can download Anaconda for free from this link. Please remember to choose the correct distribution and architecture. Below image shows Windows 64 bit Python 3.7 download link and minimum version required by Tabpy is Python 3.6.

Once the installer  is downloaded,  double click on it to start the installation. Default options are fine and the installation process will take about 10 minutes.

2. Creating Virtual Environment

From start menu, Search for Anaconda Prompt and open it. Execute following commands in order to create a virtual environment called ‘tpy’ and then to activate it.

conda create -n tpy python=3.7
activate tpy

3. Installing TabPy

From the activated ‘tpy’ environment, execute following commands:

pip install tabpy
pip install pandas
tabpy

The first two lines will install TabPy and pandas (required for Prep) and the third line will start the tabpy server at port 9004.

4. Tableau Prep Python Integration

In Order to use Python code in the flow, we need to use ‘Add Script’ step. We will use a simple function to sort the output data in descending order of sales. Save the below lines of code in a file with .py extension.

def sort_descending(df):
    """Sorts the dataframe in descending order of sales."""
    return df.sort_values(by=['Sales'], ascending=False)

Below screen grab shows the entire process. You can download the packaged flow file here.

5. TabPy Additional Configurations

TabPy is highly configurable and the configurations can be controlled using a configuration file. Common configuration settings include Changing port number, enabling HTTPS, Enabling Authentication, Change Logging Levels, etc.

Download this sample configuration file (use right click and save link as) and you can start TabPy server using the configuration using below command:

tabpy --config=path_to_the_downloaded_file

Describing all configuration settings are beyond the scope of this post, but you can find details at this link.

If you have any questions, please feel free to ask in the comments and I will try my best to answer.

The post Tableau Prep Python Integration – Installing and Setting Up TabPy appeared first on Techno Whisp.

Before we start, I have to admit that this is a complicated flow for a simple problem. If we had a concatenate function in Tableau Prep aggregation step, we could have done this reshaping in a single step. There is already an idea in the Ideas Forum. Please upvote if you like this idea.

Requirement

Data is in tall format which is analysis friendly, but we need to generate an output in csv which has one row per user and all his/her workdays are separated by comma. This output is an input to another application.

This solution includes 5 Steps:

1. Create Row IDs per group

In our case we need to create row Ids per employee. Please see this post about creating row Ids if you are having difficulty in generating row Ids per employee.

2. Convert Row Ids to Field Names

Use a calculation to convert RowIDs like 1,2,3 to F1,F2,F3. We will use this column as field names later.

3. Union with a dummy File to make field names consistent

4. Pivot Rows To columns

Add a pivot step and select row to columns. Drag ROW_ID in to the pivoted fields section and Workday to aggregate section. Use max as aggregation.

5. Concatenate Fields and Clean

Create a field called ‘Work Days’ with below formula  and keep only the required fields. Depending on your dataset, you might need to add more Fields in to your calculation. I am using 5 Fields(f1 to F5) here and the result is an aggregate string field.

Please download the packaged flowfile to play around with this flow.

The post Tableau Prep – Aggregating Strings appeared first on Techno Whisp.

Click on the ‘Group  Category’ step and you can see that there are two ungrouped categories as mentioned in the email trigger.

There are many ways to include these new categories in to our existing groups, but none of them are perfect.

Manual Grouping

Since we only have a couple of ungrouped items in the flow, this option will work for us, but it won’t be effective if the number of new items are huge.

Algorithmic Grouping

We can try to use one of the built in algorithms like ‘Spelling’ or pronunciation, but it will reset our existing grouping.

It is ok if our existing grouping is entirely based on a built in algorithm, but what if we fine tuned the results manually? We will have to redo all those optimizations again. It would have been  a lot better if Tableau  Prep allowed us to keep the existing groups, and run the algorithm to find new matches.

Nested Algorithmic Grouping

You can keep the existing  groups and run  the grouping algorithm again on top of the existing grouping. To do that, add a new group and replace in the existing ‘Group Category’ step. Please note that we are not editing the existing ‘Group and Replace’ step, but adding a new one.

While this is the easiest way to integrate new changes when large number of values to be grouped, we are losing flow readability.  How will the changes pane look after 10 iterations? How do you know which step replaces what

So which method should you choose? Well it depends a lot on personal preference, buy here is my take on this.

Manual – When I only have a couple of new values to be grouped

Nested Algorithmic Grouping – When I just want a quick fix.

Algorithmic Grouping – When I want to refractor the flow for readability. In that case I will remove all groupings and  regroup everything in a single step.

Summary

This was the final post in my Future Proofing Group And Replace Series. SO here is The steps involved:

1. Remove ungrouped(dirty) data from output
2. Generate an email trigger when ungrouped(dirty) data is detected
3. Modify the flow to handle the ungrouped(dirty) data

The post Tableau Prep – Iterating Group And Replace appeared first on Techno Whisp.

Install the utility and add a FTP folder as local drive by entering the below command in an elevated command prompt.

FTPUSE DRIVE_LETTER: yourserver.domain.com/path password /USER:username

Now the  FTP folder appears as a network drive in our PC. Please note that the drive will be disconnected on reboot. (We will remedy this in our flow schedule BAT file.)

This mapping works like a normal network drive in PREP. Since the mapping is not permanent, it is safer to remap the drive before running the flow . Here is the complete BAT file I used for running the flow.

@echo off
set prep="C:\Program Files\Tableau\Tableau Prep Builder 2019.1\scripts\tableau-prep-cli.bat"
set flow="C:\Automation\FTP_TEST.tfl"
if exist X:\ (
	echo Disconnecting existing mapping....
	ftpuse X: /delete
)
echo Mapping FTP folder to network drive....
ftpuse X: yourserver/path password /USER:username
call  %prep% -t %flow%
PAUSE

Here is a screen grab of the flow run.

The post How to connect to FTP with Tableau Prep appeared first on Techno Whisp.

Unfortunately Tableau Prep does not have a built in alert feature yet, so we have to depend on Tableau Server. Change the flow in our last post to write the ungrouped data in to Tableau server

Now  let’s create a simple chart that shows the count of the categories as below and publish it to tableau server.

Go to Tableau server and open the view. Create an alert for “Number Of Records” Above or Equal To One. If there are no ungrouped categories, our chart will be empty. We will receive a trigger as long as there is at least one ungrouped category detected in the flow.

Now play with different input files, you will receive an email if the flow generate any ungrouped categories. (You might need to change the alert frequency during testing as the setting shown will trigger only one email per day).

The post Tableau Prep Email Alerts appeared first on Techno Whisp.

1. Remove ungrouped(dirty) data from output
2. Generate an email trigger when ungrouped(dirty) data is detected
3. Modify the flow to handle the ungrouped(dirty) data

We will address the first item in this post and  the remaining in near future. We will start with the same flow and data set in last post.

Step 1

Create a list of master categories we want to maintain(Office Supplies, Furniture and Technology). You can use Excel to create this list or a flow like below will output only the master categories to the output csv file.

Step 2

Now bring in the master category in to the flow and insert a left join as shown below. All data in the join will mach now, but any future ungrouped/dirty category will have null values at the right side of this join and we can use this fact to filter them out.

Step 3

Insert a clean step after the join and keep only ‘Non-null ‘ values  in category-1(Master_Category) filter. You can also remove this field as we dont need it in the output.

Step 4

Now, we need to identify the ungrouped/dirty categories. Insert a branch after join and keep only ‘Null’ values in Category-1(Master Category). This will be empty for now as we dont have any ungrouped data, but we are future proofing our flow, remember?

Step 5

Our intention is to identify the ungrouped categories in our dataset. Add an aggregation after the clean step and group by categories and write it to an output file, we have our final flow.

Testing The Flow

Add some additional rows to the input data with new categories and run the flow.(Alternatively you can use this file). You can  see that the cleaned data output only  have 3 categories as expected. Additionally, ungrouped categories are added to Category_Ungroup.csv.

Next Steps

1. Generate an email trigger when ungrouped(dirty) data is detected
2. Modify the flow to handle the ungrouped(dirty) data

The post Tableau Prep – Future Proofing Group and Replace appeared first on Techno Whisp.

1. Algorithms are applied only to the sampled data and not to the entire data set
   • Prep flow stores a look up table of replacements  based on sampled data and applies that look up table to the entire data set when running the flow
   • This is justifiable as  grouping  applied to unseen combinations can be a bad thing. For example,  we do not want  Prep’s common character grouping algorithm to group words with same characters like ‘God’ and ‘Dog’ together.
2. It is extremely difficult to get the sampling right for huge data sets.
   • Default sampling limit in Prep is 1 million rows, but it may or may not be enough depending on the size and variety of  the  data set.
   • There is a great post on Tableau’s blog about improving the sampling efficiency
3. Multiple iterations are required to get the group and replace right due to above issues

But how can we do this effectively without sampling all data or performing multiple iterations?  I will provide the steps  below, but  here is the data set to clean if you would like to attempt the challenge yourselves.

Rules:

1. Final data set should group the existing categories in to 3 categories(Office Supplies, Furniture and Technology).
2. You are allowed to sample only up to 50% the data- up to 5 rows in this case  (This is to simulate real life scenarios where the data size does not allow to sample entire data set)
3. You are not allowed to manually add a category that doesn’t exists in the data sample. (This wont be practical in large data sets with millions of rows)

Have you tried it? How did it go? Please share your solution in comments. I am sharing  the steps of my solution below:

1.  Open the data file and set the sampling size to one. (Yes, we only need one row)
2. Create an aggregate step and drag category to grouped fields.  Write it to a csv by creating an output step. This will create a file with all unique categories in the original file
3. Union the above category file with the flow and perform group and replace as normal for category field
4. Create an output step after group and replace and delete all unnecessary steps
5. Run the Flow

Please refer to below screen grab for detailed illustration. (You can click on the image to open it in a new window).

Please note that multiple columns can be added to the aggregation in step 2 if you are planning to apply group and replace to multiple fields.

The post Tableau Prep Tip – Getting Group And Replace Right At One Go For Enormous Data Sets appeared first on Techno Whisp.

We will be using  IEX Share price API for the purpose of this demonstration. Try the below one liner in your command tine and see the results. (This end point provides last 3 month share prices of Apple).

powershell.exe "Invoke-RestMethod 'https://api.iextrading.com/1.0/stock/aapl/chart/3m'"

If your rest endpoint requires authentication or custom headers, ‘Invoke-RestMethod’  cmdlet have options to handle that too.

Here is the complete script to download data, convert to csv and run the flow

@echo off
powershell.exe "(Invoke-RestMethod 'https://api.iextrading.com/1.0/stock/aapl/chart/3m')|Export-Csv -Path 'C:/Automation/rest_data.csv' -NoTypeInformation"
call  "C:\Program Files\Tableau\Tableau Prep 2019.1\scripts\tableau-prep-cli.bat" -t "C:\Automation\rest.tfl"
PAUSE

The post Tableau Prep – Connecting to REST API appeared first on Techno Whisp.