Data

Transformation

Model selection

The anomalous package provides some tools to detect unusual time series in a large collection of time series. This is joint work with Earo Wang (an honours student at Monash) and Nikolay Laptev (from Yahoo Labs). Yahoo is interested in detecting unusual patterns in server metrics.

The basic idea is to measure a range of features of the time series (such as strength of seasonality, an index of spikiness, first order autocorrelation, etc.) Then a principal component decomposition of the feature matrix is calculated, and outliers are identified in 2-dimensional space of the first two principal component scores.

We use two methods to identify outliers.

1. A bivariate kernel density estimate of the first two PC scores is computed, and the points are ordered based on the value of the density at each observation. This gives us a ranking of most outlying (least density) to least outlying (highest density).
2. A series of –convex hulls are computed on the first two PC scores with decreasing , and points are classified as outliers when they become singletons separated from the main hull. This gives us an alternative ranking with the most outlying having separated at the highest value of , and the remaining outliers with decreasing values of .

I explained the ideas in a talk last Tuesday given at a joint meeting of the Statistical Society of Australia and the Melbourne Data Science Meetup Group. Slides are available here. A link to a video of the talk will also be added there when it is ready.

The density-ranking of PC scores was also used in my work on detecting outliers in functional data. See my 2010 JCGS paper and the associated rainbow package for R.

There are two versions of the package: one under an ACM licence, and a limited version under a GPL licence. Eventually we hope to make the GPL version contain everything, but we are currently dependent on the alphahull package which has an ACM licence.

After a few quiet months, a new version of rfoaas is now on CRAN. As before, it shadows upstream release of FOAAS from which carried over the version number 0.1.6.

The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off.

Release 0.1.6 of FOAAS builds on the initial support for filters and now adds working internationalization. This is best shown by example:

R> library(rfoaas)
R> off("Tom", "Everyone")                                      # standard upstream example
[1] "Fuck off, Tom. - Everyone"
R> off("Tom", "Everyone", language="fr")                       # now in French
[1] "Va te faire foutre, Tom. - Everyone"
R> off("Tom", "Everyone", language="de", filter="shoutcloud")  # or in German and LOUD
[1] "VERPISS DICH, TOM. - EVERYONE"
R> 

We start with a standard call to off(), add the (now fully functional upstream) language support and finally illustrate the shoutcloud.io filter added in the preceding 0.1.3 release.

As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

About a month ago I made an announcement about the initial release of shinyjs. After some feedback, a few feature requests, and numerous hours of work, I’m excited to say that a new version of shinyjs v0.0.6.2 was made available on CRAN this week. The package’s main objective is to make shiny app development better and easier by allowing you to perform many useful functions with simple R code that would normally require JavaScript coding. Some of the features include hiding/showing elements, enabling/disabling inputs, resetting an input to its original value, and many others.

Table of contents

• Availability
• Quick overview of new features
• Major new features
  • reset - allows inputs to be reset to their original value
  • extendshinyjs - allows you to easily call your own JavaScript functions from R
• Major improvements
  • Enabling/disabling works on all inputs
  • Use a condition in toggle functions
• Extra features on GitHub but not yet on CRAN
  • hidden now accepts multiple tags
  • Visibility functions can be run on any selector
  • Visibility functions can be delayed
• Feedback + suggestions

Availability

shinyjs is available through both CRAN (install.packages("shinyjs")) and GitHub (devtools::install_github("daattali/shinyjs")). Use the GitHub version to get the latest version with the newest features.

Quick overview of new features

This post will only discuss new features in shinyjs. You can find out more about the package in the initial post or in the package README on GitHub. Remember that in order to use any function, you need to add a call to useShinyjs() in the shiny app’s UI.

Two major new features:

• reset function allows inputs to be reset to their original value
• extendShinyjs allows you to add your own JavaScript functions and easily call them from R as regular R code

Two major improvements:

• Enabling and disabling of input widgets now works on all types of shiny inputs (many people asked how to disable a slider/select input/date range/etc, and shinyjs now handles all of them)
• The toggle functions gained an additional condition argument, which can be used to show/hide or enable/disable an element based on a condition. For example, instead of writing code such as if (test) enable(id) else disable(id), you can simply write toggleState(id, test)

Three new features available on the GitHub version but not yet on CRAN:

• hidden (used to initialize a shiny tag as hidden) can now accept any number of tags or a tagList rather than just a single tag
• hide/show/toggle can be run on any JQuery selector, not only on a single ID, so that you can hide multiple elements simultaneously
• hide/show/toggle have a new arugment delay which can be used to perform the action later rather than immediately. This can be useful if you want to show a message and have it disappear after a few seconds

Two major new features

There were two major features that I wanted to include in the CRAN release.

reset - allows inputs to be reset to their original value

Being able to reset the value of an input has been a frequently asked question on StackOverflow and the shiny Google Group, but a general solution was never available. Now with shinyjs it’s possible and very easy: if you have an input with id name and you want to reset it to its original value, simply call reset("name"). It doesn’t matter what type of input it is - reset works with all shiny inputs.

The reset function only takes one arugment, an HTML id, and resets all inputs inside of that element. This makes reset very flexible because you can either give it a single input widget to reset, or a form that contains many inputs and reset them all. Note that reset can only work on inputs that are generated from the app’s ui and it will not work for inputs generated dynamically using uiOutput/renderUI.

Here is a simple demo of reset in action

extendShinyjs - allows you to easily call your own JavaScript functions from R

The main idea behind shinyjs when I started working on it was to make it extremely easy to call JavaScript functions that I used commonly from R. Now whenever I want to add a new function to shinyjs (such as the reset function), all I have to do is write the JavaScript function, and the integration between shiny and JavaScript happens seamlessly thanks to shinyjs. My main goal after the initial release was to also allow anyone else to use the same smooth R –> JS workflow, so that anyone can add a JavaScript function and call it from R easily. With the extendShinyjs function, that is now possible.

Very simple example

Using extendShinyjs is very simple and makes defining and calling JavaScript functions painless. Here is a very basic example of using extendShinyjs to define a (fairly useless) function that changes the colour of the page.

library(shiny)
library(shinyjs)

jsCode <- "shinyjs.pageCol = function(params){$('body').css('background', params);}"

runApp(shinyApp(
  ui = fluidPage(
    useShinyjs(),
    extendShinyjs(text = jsCode),
    selectInput("col", "Colour:",
                c("white", "yellow", "red", "blue", "purple"))
  ),
  server = function(input,output,session) {
    observeEvent(input$col, {
      js$pageCol(input$col)
    })
  }
))

Running the code above produces this shiny app:

See how easy that was? All I had to do was make the JavaScript function shinyjs.pageCol, pass the JavaScript code as an argument to extendShinyjs, and then I can call js$pageCol(). That’s the basic idea: any JavaScript function named shinyjs.foo will be available to call as js$foo(). You can either pass the JS code as a string to the text argument, or place the JS code in a separate JavaScript file and use the script argument to specify where the code can be found. Using a separate file is generally prefered over writing the code inline, but in these examples I will use the text argument to keep it simple.

Passing arguments from R to JavaScript

Any shinyjs function that is called will pass a single array-like parameter to its corresponding JavaScript function. If the function in R was called with unnamed arguments, then it will pass an Array of the arguments; if the R arguments are named then it will pass an Object with key-value pairs. For example, calling js$foo("bar", 5) in R will call shinyjs.foo(["bar", 5]) in JS, while calling js$foo(num = 5, id = "bar") in R will call shinyjs.foo({num : 5, id : "bar"}) in JS. This means the shinyjs.foo function needs to be able to deal with both types of parameters.

To assist in normalizing the parameters, shinyjs provides a shinyjs.getParams() function which serves two purposes. First of all, it ensures that all arguments are named (even if the R function was called without names). Secondly, it allows you to define default values for arguments. Here is an example of a JS function that changes the background colour of an element and uses shinyjs.getParams().

shinyjs.backgroundCol = function(params) {
  var defaultParams = {
    id : null,
    col : "red"
  };
  params = shinyjs.getParams(params, defaultParams);

  var el = $("#" + params.id);
  el.css("background-color", params.col);
}

Note the defaultParams that we defined and the call to shinyjs.getParams. It ensures that calling js$backgroundCol("test", "blue") and js$backgroundCol(id = "test", col = "blue") and js$backgroundCol(col = "blue", id = "test") are all equivalent, and that if the colour parameter is not provided then “red” will be the default. All the functions provided in shinyjs make use of shinyjs.getParams, and it is highly recommended to always use it in your functions as well. Notice that the order of the arguments in defaultParams in the JavaScript function matches the order of the arguments when calling the function in R with unnamed arguments. This means that js$backgroundCol("blue", "test") will not work because the arguments are unnamed and the JS function expects the id to come before the colour.

For completeness, here is the code for a shiny app that uses the above function (it’s not a very practical example, but it’s great for showing how to use extendShinyjs with parameters):

library(shiny)
library(shinyjs)

jsCode <- '
shinyjs.backgroundCol = function(params) {
  var defaultParams = {
    id : null,
    col : "red"
  };
  params = shinyjs.getParams(params, defaultParams);

  var el = $("#" + params.id);
  el.css("background-color", params.col);
}'

runApp(shinyApp(
  ui = fluidPage(
    useShinyjs(),
    extendShinyjs(text = jsCode),
    p(id = "name", "My name is Dean"),
    p(id = "sport", "I like soccer"),
    selectInput("col", "Colour:",
                c("white", "yellow", "red", "blue", "purple")),    
    textInput("selector", "Element", ""),
    actionButton("btn", "Go")
  ),
  server = function(input,output,session) {
    observeEvent(input$btn, {
      js$backgroundCol(input$selector, input$col)
    })
  }
))

And the resulting app:

Note that I chose to define the JS code as a string for illustration purposes, but in reality I would prefer to place the code in a separate file and use the script argument instead of text.

Two major improvements

Among the many small improvements made, there are two that will be the most useful.

Enabling/disabling works on all inputs

The initial release of shinyjs had a disable/enable function which worked on the major input types that I was commonly using, but not all. Several people noticed that various inputs could not be disabled, so I made sure to fix all of them in the next version. The reasons behind why not all inputs were easy to disable are very technical so I won’t go into them. Now calling disable(id) or enable(id) will work on any type of shiny input.

Use a condition in toggle functions

I’ve noticed that some users of shinyjs had to often write code such as if (test) enable(id) else disable(id). This seemed inefficient and verbose, especially since there was already a toggleState function that enables disabled elements and vice versa. The toggleState function now has a new condition parameter to address exactly this problem. The code above can now be rewritten as toggleState(id, test).

Similarly, code that previously used if (test) show(id) else hide(id) can now use toggle(id = id, condition = test), and code that was doing a similar thing with addClass/removeClass can use the toggleClass(id, class, condition) function.

Three new features available on the GitHub version but not yet on CRAN

Since submitting shinyjs to CRAN, there were a few more features added. They will go into the next CRAN submission in about a month, but for now they can be used if you download the GitHub version.

hidden now accepts multiple tags

The hidden function is the only shinyjs function that’s used in the UI rather than in the server. It’s used to initialize a tag as hidden, and you can later reveal it using show(tag). The initial release only allows single tags to be given to hidden, but now it can accept any number of tags or a tagList. For example, you can now add a button and some text to the UI and have them both hidden with this code: hidden(actionButton("btn", "Button"), p(id = "text", "text")). You can then call show("btn") or show("text") to unhide them.

Visibility functions can be run on any selector

Previously, the only way to tell the hide, show, and toggle functions what element to act on was to give them an ID. That becomes very limiting when you want to hide or show elements in batch, or even if you just want to show/hide an element without an ID. The visibility functions now have a new optional parameter selector that accepts any CSS-style selector. For example, to hide all hyperlinks on a page that have the class “hide-me-later” you can now call hide(selector = "a.hide-me-later"). This makes the visibility functions much more powerful.

Visibility functions can be delayed

In a shiny app that I’m currently developing for my graduate work there are many different “Update” buttons that the user can click on. After an update is successful, I wanted to show a “Done!” message that would disappear after a few seconds. Using shinyjs I was already able to show the message when I wanted to, but I needed an easy way to make it disappear later. So I added the delay parameter to show/hide/toggle, that tells the function to only act in x seconds instead of immediately. Now if I want to show a message and hide it after 5 seconds, I can call show("doneMsg"); hide(id = "doneMsg", delay = 5). It’s not a big deal, but it can be handy.

Feedback + suggestions

If you have any feedback on shinyjs, I’d love to hear about it! I really do hope that it’s as easy to use as possible and that many of you will find it useful. If you have any suggestions, please do open a GitHub issue or let me know in any other way.

The term “mail merge” might not be familiar to those who have not worked in an office setting, but here is the Wikipedia definition:

Mail merge is a software operation describing the production of multiple (and potentially large numbers of) documents from a single template form and a structured data source. The letter may be sent out to many “recipients” with small changes, such as a change of address or a change in the greeting line.

Source: http://en.wikipedia.org/wiki/Mail_merge

The other day I was working on creating personalized handouts for a workshop. That is, each handout contained some standard text (including some R code) and some fields that were personalized for each participant (login information for our RStudio server). I wanted to do this in RMarkdown so that the R code on the handout could be formatted nicely. Googling “rmarkdown mail merge” didn’t yield much (that’s why I’m posting this), but I finally came across this tutorial which called the process “iterative reporting”.

Turns our this is a pretty straightforward task. Below is a very simple minimum working example. You can obviously make your markdown document a lot more complicated. I’m thinking holiday cards made in R…

All relevant files for this example can also be found here.

Input data: meeting_times.csv

This is a 20 x 2 csv file, an excerpt is shown below. I got the names from here.

R script: mail_merge_script.R

## Packages
library(knitr)
library(rmarkdown)

## Data
personalized_info <- read.csv(file = "meeting_times.csv")

## Loop
for (i in 1:nrow(personalized_info)){
 rmarkdown::render(input = "mail_merge_handout.Rmd",
 output_format = "pdf_document",
 output_file = paste("handout_", i, ".pdf", sep=''),
 output_dir = "handouts/")
}

RMarkdown: mail_merge_handout.Rmd

---
output: pdf_document
---

```{r echo=FALSE}
personalized_info <- read.csv("meeting_times.csv", stringsAsFactors = FALSE)
name <- personalized_info$name[i]
time <- personalized_info$meeting_time[i]
```

Dear `r name`,

Your meeting time is `r time`.

See you then!

Save the Rmd file and the R script in the same folder (or specify the path to the Rmd file accordingly in the R script), and then run the R script. This will call the Rmd file within the loop and output 20 PDF files to the handouts directory. Each of these files look something like this

with the name and date field being different in each one.

If you prefer HTML or Word output, you can specify this in the output_format argument in the R script.

Every finite game has an equilibrium point (John Nash, Non-Cooperative Games, 1950)

I read recently this amazing book, where I discovered that we (humans) are not capable of generating random sequences of numbers by ourselves when we play lottery. John Haigh demonstrates this fact analyzing a sample of 282 raffles of 6/49 UK Lotto. Once I read this, I decided to prove if this disability is property only of British population or if it is shared with Spanish people as well. I am Spanish, so this experiment can bring painful results to myself, but here I come.

The Spanish equivalent of 6/40 UK Lotto is called “Lotería Primitiva” (or “Primitiva”, to abbreviate). This is a ticket of Primitiva lotto:

As you can see, one ticket gives the chance to do 8 bets. Each bet consists on 6 numbers between 1 and 49 to be chosen in a grid of 10 rows by 5 columns. People tend to choose separate numbers because we think that they are more likely to come up than combinations with some consecutive numbers. We think we have more chances to get rich choosing 4-12-23-25-31-43 rather than 3-17-18-19-32-33, for instance. To be honest, I should recognize I am one of these persons.

Primitiva lotto is managed by Sociedad Estatal Loterías y Apuestas del Estado, a public business entity belonging to the Spanish Ministry of Finance and Public Administrations. They know what people choose and they could do this experiment more exactly than me. They could analyze just human bets (those made by players by themselves) and discard machine ones (those made automatically by vending machines) but anyway it is possible to confirm the previous thesis with some public data.

I analysed 432 raffles of Primitiva carried out between 2011 and 2015; for each raffle I have this information:

• The six numbers that form the winning combination
• Total number of bets
• Number of bets which hit the six numbers (Observed Winners)

The idea is to compare observed winners of raffles with the expected number of them, estimated as follows:

This table compare the number of expected and observed winners between raffles which contain consecutive and raffles which not:

There are 214 raffles without consecutive with 294 winners while the expected number of them was 219. In other words, a winner of a non-consecutive-raffle must share the prize with a 33% of some other person. On the other hand, the number of observed winners of a raffle with consecutive numbers 17% lower than the expected one. Simple and conclusive. Spanish are like British, at least in what concerns to this particular issue.

Let’s go further. I can do the same for any particular number. For example, there were 63 raffles containing number 45 in the winning combination and 57 (observed) winners, although 66 were expected. After doing this for every number, I can draw this plot, where I paint in blue those which ratio of observed winners between expected is lower than 0.9:

It seems that blue numbers are concentrated on the right side of the grid. Do we prefer small numbers rather than big ones? There are 15 primes between 1 and 49 (rate: 30%) but only 3 primes between blue numbers (rate: 23%). Are we attracted by primes?

Let’s combine both previous results. This table compares the number of expected and observed winners between raffles which contain consecutive and blues (at least one) and raffles which not:

Now, winning combinations with some consecutive and some blue numbers present 20% less of observed winners than expected. After this, which combination would you choose for your next bet? 27-35-36-41-44-45 or 2-6-13-15-26-28? I would choose the first one. Both of them have the same probability to come up, but probably you will become richer with the first one if it happens.

This is the code of this experiment. If someone need the dataset set to do their own experiments, feel free to ask me (you can find my email here):

library("xlsx")  
library("sqldf")
library("Hmisc")
library("lubridate")
library("ggplot2")
library("extrafont")
library("googleVis")
windowsFonts(Garamond=windowsFont("Garamond"))
setwd("YOUR WORKING DIRECTORY HERE")
file = "SORTEOS_PRIMITIVA_2011_2015.xls"
data=read.xlsx(file, sheetName="ALL", colClasses=c("numeric", "Date", rep("numeric", 21)))  
#Impute null values to zero
data$C1_EUROS=with(data, impute(C1_EUROS, 0))
data$CE_WINNERS=with(data, impute(CE_WINNERS, 0))
#Expected winners for each raffle
data$EXPECTED=data$BETS/(factorial(49)/(factorial(49-6)*factorial(6)))
#Consecutives indicator
data$DIFFMIN=apply(data[,3:8], 1, function (x) min(diff(sort(x))))
#Consecutives vs non-consecutives comparison
df1=sqldf("SELECT CASE WHEN DIFFMIN=1 THEN 'Yes' ELSE 'No' END AS CONS, 
      COUNT(*) AS RAFFLES,
      SUM(EXPECTED) AS EXP_WINNERS, 
      SUM(CE_WINNERS+C1_WINNERS) AS OBS_WINNERS
      FROM data GROUP BY CONS")
colnames(df1)=c("Contains consecutives?", "Number of  raffles", "Expected Winners", "Observed Winners")
Table1=gvisTable(df1, formats=list('Expected Winners'='#,###'))
plot(Table1)
#Heat map of each number
results=data.frame(BALL=numeric(0), EXP_WINNER=numeric(0), OBS_WINNERS=numeric(0))
for (i in 1:49)
{
  data$TF=apply(data[,3:8], 1, function (x) i %in% x + 0)
  v=data.frame(BALL=i, sqldf("SELECT SUM(EXPECTED) AS EXP_WINNERS, SUM(CE_WINNERS+C1_WINNERS) AS OBS_WINNERS FROM data WHERE TF = 1"))
  results=rbind(results, v)
}
results$ObsByExp=results$OBS_WINNERS/results$EXP_WINNERS
results$ROW=results$BALL%%10+1
results$COL=floor(results$BALL/10)+1
results$ObsByExp2=with(results, cut(ObsByExp, breaks=c(-Inf,.9,Inf), right = FALSE))
opt=theme(legend.position="none",
          panel.background = element_blank(),
          panel.grid = element_blank(),
          axis.ticks=element_blank(),
          axis.title=element_blank(),
          axis.text =element_blank())
ggplot(results, aes(y=ROW, x=COL)) +
  geom_tile(aes(fill = ObsByExp2), colour="gray85", lwd=2) +
  geom_text(aes(family="Garamond"), label=results$BALL, color="gray10", size=12)+
  scale_fill_manual(values = c("dodgerblue", "gray98"))+
  scale_y_reverse()+opt
#Blue numbers
Bl=subset(results, ObsByExp2=="[-Inf,0.9)")[,1]
data$BLUES=apply(data[,3:8], 1, function (x) length(intersect(x,Bl)))
#Combination of consecutives and blues
df2=sqldf("SELECT CASE WHEN DIFFMIN=1 AND BLUES>0 THEN 'Yes' ELSE 'No' END AS IND, 
      COUNT(*) AS RAFFLES,
      SUM(EXPECTED) AS EXP_WINNERS, 
      SUM(CE_WINNERS+C1_WINNERS) AS OBS_WINNERS
      FROM data GROUP BY IND")
colnames(df2)=c("Contains consecutives and blues?", "Number of  raffles", "Expected Winners", "Observed Winners")
Table2=gvisTable(df2, formats=list('Expected Winners'='#,###'))
plot(Table2)

By Douglas Ashton, Consultant

One of the big successes of data analytics is the cultural change in how business decisions are being made. There is now wide spread acceptance of the role that data science has to play in decision making. With the explosion in the quantity of data available, the task for the modern analyst is to filter through to find the information that is most relevant.

Twitter represents a classic case. The volume and velocity of Twitter data is staggering, and as discussed in the first part of this series it is within reach to obtain large, clean datasets. This puts the pressure on the analyst to ask the right questions of the data. In the remaining parts of this series we’ll be looking through a variety of ways to view the data from Twitter. The view you wish to take will ultimately depend on the question that is being asked. In this post we’re tackling the question: “Who are the big players in the data science Twitter community?”.

The dataset that we will be using is a sample of all tweets tagged with the hashtags #datascience, #rstats and #python (snake related tweets cleaned out) between the 7th and 17th December 2014. Each tweet contains a number of pieces of useful information. The view that we’re going to take in this post is the mentions network.

Mentions Network

There are a number of ways users can interact on Twitter. Users can “follow” other users to receive regular updates of their tweets, and users may “mention” other users in their own tweets.

In the tweet above I mentioned @MangoTheCat in the text of the tweet and so we draw a directed link from me, @dougashton to @MangoTheCat. A retweet is another way for one user to mention another. We then went through each of the 22545 tweets in our data set and formed links for every mention. The resulting network contained 8220 nodes (users) and 20461 edges (mentions). In network language an “edge” is the same as a link.

A useful tool for dealing with networks in R is the feature rich igraph package (also available for Python and C). Once you have created your network as an igraph object many of the standard network analysis tools become easily available.

While igraph has nice built-in plotting tools, for large graphs I also like the cross-platform, open source, Gephi. Gephi is an interactive network visualisation and analysis tool with many available plugins. We can easily export our graph from igraph to an open format, such as graphml, and read it into Gephi. The layout below shows the full mentions network and was created using the Force Atlas 2 layout.

Even with the limited sample that we used, this is a big network. This visualisation is useful as a high level view of the network. For instance, we have a connected core of tweeters and a disconnected periphery, which is to be expected with this type of sampling technique. We can also see a common motif of clusters of nodes around one other node, these look like parachutes in the visualisation. To gain further insight we must dig a little deeper into the data.

Broadcasters and Receivers

As noted above, this network appears to contain some nodes that are surrounded by large cluster of other nodes. We can quantify this by looking at the degree centrality. The in-degree of a node is the total number of tweets that mention that user. Similarly the out-degree is the number of mentions made by that user. The top ten nodes for both in and out degree are listed below:

The two lists are completely different. We have a group of users with a large out-degree who retweet at a high rate. These are our broadcasters. In general they tend to pass on content. The users with a large in-degree have their tweets retweeted many times by different users. These are our receivers.

It might be that you can stop there. The nodes with a high in-degree are likely important nodes in this network. However, how do we really know how far their influence really goes? It is possible to get a high in-degree score by being retweeted many times by a small group of broadcasters. This is not a guarantee of influence. For a broader view we must go beyond this nearest neighbour approach and look to more sophisticated network measures.

Centrality Measures

There are many standard measures of network structure available and the choice of which one to use really comes down to exactly what you are interested in. Here I’ll go through two, Page Rank and Betweenness Centrality. In igraph it’s as easy as running

Page Rank is the basis of Google’s search engine. Roughly speaking it tells you which nodes you are likely to land on if you spent some time surfing twitter feeds randomly following links. If mentions tend to flow towards you, you will get a high score. It also shows that you are connected to other influential nodes.

Betweenness Centrality is a useful measure for networks with a strong community structure. If you work out all of the shortest paths between all of the nodes, betweenness tells you how many of those paths go through each node. If you are the bridge between two communities then you will get a high score.

This time we see many of the same users but now some familiar names begin to appear in these lists. For instance, we know that @hadleywickham is an influential figure in the R community, and while Hadley only tweeted 23 times in this period he features high up the Page Rank centrality list.

@MangoTheCat only tweeted four times in this period yet the cat’s betweenness score is relatively high. This implies that the connections formed in those tweets were bridging connections between different types of node. We can see this a little better if we look at a much smaller version of our network. The strongly connected component is the part of the network where you can travel from any node to any node along the links. We get this by finding the clusters and keeping the largest.

While a little out from the core we see that @MangoTheCat does indeed sit between the core and a group on the edge of the cluster.

Summing up

We’ve seen that the tools available in R have made aquiring and analysing the Twitter network an easily accessible task. In this post we’ve dipped our toe into the vast array of methods available in network analysis and we’ve found that digging a little deeper than simply counting connections can lead to a deeper insight of the network’s true function. With so much data available it is important to ask the right questions. If your goal is to improve your social media presence, or whether you are trying to better search for influential users, a little bit of the right network tools can go a long way.

High-dimensional Euclidean space is ℝ×ℝ×ℝ×ℝ×ℝ×…. Cartesian product of many continuous quantities.

You are already familiar with the concept via “an arbitrary number of sliders” or “an arbitrary number of columns in a spreadsheet”. Pivot tables are the normal way business people navigate a low-dimensional subset of a high-dimensional space.

One database or slider-config would be one point in a high-dimensional space. But what do these spaces look like overall?

Lots of Corners

Think about the graph-skeleton of cubes and squares. Zoom in on a corner.

Look at how the number of neighbours increases as you increase dimension. The corner of a box in 2-D has two sticks coming out of it; in 3-D three sticks come out of it. In 30 dimensions there would be 30 sticks coming out of each ball. (Toward the interior. You’d need 2•30 sticks if we’re talking about a grid rather than a box.)

More on the outside, less on the inside.

Think about a Matryushka of nested spheres ⬤⊂⬤⊂⬤⊂… or, ok, even dolls.

The 10th one requires more plastic than the 1st one.

Picture by Borb. Licensed under CC BY-SA 3.0 via Wikimedia Commons.

In 3-D a two-dimensional surface gets larger as radius². In 30 dimensions surface areas will go up like radius29.

Fatter at the equator.

Circles of latitude are wider in the middle of the ball than at the top.

But this is a consequence of the coördinatisation (lat,long), not of the object itself. A hectare in Finland is as big as a hectare in Kenya. And I’m not suddenly faster running in Finland either.

(Manifolds solve the problem by patching together different charts, each of which shows one area more faithfully.)

The sphere is tiny or the cube’s mass is in its spikes.

This is justified by comparing spheres with cubes.

In the 3-D the corners of the cube stick out beyond the sphere. I could measure the size of these corners by putting another sphere outside the corners, and calculating volume(circumsphere) − volume(insphere). The Earth has more volume in its mantle than in its inner core.

Combine this with what I said above in the Beatles Matryushka section. I’m seeing the mantle and core as successions of nested shells of volume ≈ ∫radius², each bigger than the last. Once again this pattern will only get more extreme in high dimensions.

Simplices

These are the spaces on which probability happens. In 3 variables I can think about (⅓,⅓,⅓), (⅔,0,⅓), (0,1,0), and (0,0,1).

Do they have to get more complicated when we pad them with zeroes? (0,0,⅓,0,0,⅓,0,0,0,⅓) and (0,0,⅔,0,0,0,0,0,0,⅓) tell me that a 10-simplex has 10 choose 3 = 120 3-simplices embedded in it. But (.1,.1,.1,.1,.1,.1,.1,.1,.1,.1) tells me it contains more than that.

I can still think about the 10-simplex as connections between (1,0,0,0,0,0,0,0,0,0), (0,1,0,0,0,0,0,0,0,0), … (0,0,0,0,0,0,0,0,0,1), but now there are more ways to connect—since each of those points has up to 9 buddies it can be heading toward at a time.

When 2 entries are nonzero, that’s a 2-D surface. When 7 entries are nonzero, that’s a 7-D surface. (Just think of 7 sliders.)

3-Spheres on Up

All of this mention of spheres, it’d be nice to have an alternative description to w²+x²+y²+z²=1 and sin θ₁ sin θ₂ sin θ₃ … cos θₙ, which I’ve always found hard to digest.

I know that S¹×S¹ isn’t S², because that would be a torus instead. So what do I combine, and how, to get S²?

I like the Drawstring Bag model:

That takes me from the open disk to S² (= the surface of the Earth—sans magma). This is a topological construction so small perturbations are ok. Calling the geoid a “sphere” is ok. The disk doesn’t have to be exactly circular.

To get to the 3-sphere I then imagine a solid sphere (= Earth + magma), attach another point ∞, and identify all of the Earth’s crust with ∞.

Going back to the open disk, here’s another imagination-exercise I do to clear this up for myself. In some of the early Mario games, a glitch you could do was to put half of Mario’s body on one side of the screen and half on the right. This is what I think of when I imagine two sides of a square being identified.

(You can play inside an octagon-with-identifications too. Thicken this octagon and imagine yourself flying around inside it. Imagine waving your arm through eg a̅₁. Where does it come out? Turn your head and watch what it’s doing. BTW, just like Mario and Pac–Man, this identification manifold was considered by entertainment artists around the same time period—remember Scooby-Doo and the Gang running in and out of hotel doors? But the animators will change the rules either over time or for different people or introduce noncommutativity—part of the humour, which proves that people do intuitively understand identification polyhedra (as well as, obviously, object permanence), or else those changes wouldn’t be funny.)

“DunceHatSpace”. Licensed under Public Domain via Wikimedia Commons.

To do this with the drawstring-bag, I imagine a circle of friends. Even though they look like they’re standing all the way across the disk from each other, they’re actually immediately next to each other at ★, and they could make a ring and touch hands and dance around it.

Or, a solitary explorer at ★ could take small steps and look, in the disk model, like he’s jumped all the way across the Earth. He stretches his hand out and, like Mario, it shows up on the other side. But this is just an aberrance of the disk model, not of S². (This also tells you how the gridding should go on the disk.)

So in symbols, T² = ◯×◯ ≠ S² = ⬤/◯ = ⬤/∂⬤ (The pattern Im ƒ / ker ƒ is much broader, including the +c of integration.)

Specific dimension sometimes matters.

A lot of what I’m walking you through is dimension → ∞ kind of stuff. But sometimes odd or even matters. And sometimes there are structures that show up in like dimension 7 or even seemingly random high numbers. Just a warning.

Computing on the surface of a high-dimensional sphere.

Persi Diaconis explains how to do this. The trick is to think of “matrices with determinant one” as the group SO(n), which represents either a point on a sphere or the action of twisting a sphere so ★ moves to that point. (The “noun” and “verb” versions are so easily interchangeable that mathematicians will usually elide the two.) If you allow determinant ±1, that’s O(n)—so allowing reflection—and O(n) is what the Gram-Schmidt algorithm reduces matrices to in order to solve [A]⋅x⃗ = b⃗.

To do what Dr Diaconis says in R you would run this 2-liner:

n = 3
rnorm(n**2) %>% matrix(n,n) %>% qr %>% qr.Q

which you can verify has determinant ±1 and is orthogonal with things like:

rnorm(n**2) %>% matrix(n,n) %>% qr %>% qr.Q -> diaconis
diaconis %>% eigen
{diaconis %>% eigen}$values %>% Mod     #eigenvalues of an O(n) are length 1
diaconis %>% det
diaconis %>% crossprod %>% zapsmall     #diaconis %*% t(diaconis)

Use rotations.

A heap of linear algebra—and therefore the logic of high-dimensional Euclidean space—is simplified if you can imagine quotienting by SO(n).

Having the drawstring-bag model and the stick+ball model and the ability to compute examples with matrices and the rectangular-grid model helps me imagine high-dimensional space. Hope that was useful for you.

Next steps

• Tits Buildings
• Coxeter-Dynkin diagrams
• Polytopes
• cell complexes

Last week the commonmark package was released on CRAN. The package implements some very thin R bindings to John Macfarlane’s (author of pandoc) cmark library. From the cmark readme:

cmark is the C reference implementation of CommonMark, a rationalized version of Markdown syntax with a spec. It provides a shared library (libcmark) with functions for parsing CommonMark documents to an abstract syntax tree (AST), manipulating the AST, and rendering the document to HTML, groff man, CommonMark, or an XML representation of the AST.

Each of the R wrapping functions parses markdown and renders it to one of the output formats:

md <- "
## Test
My list:
  - foo
  - bar"

The markdown_html function converts markdown to HTML:

library(commonmark)
cat(markdown_html(md))

<h2>Test</h2>
<p>My list:</p>
<ul>
<li>foo</li>
<li>bar</li>
</ul>

The markdown_xml function gives the parse tree in xml format:

cat(markdown_xml(md))

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE CommonMark SYSTEM "CommonMark.dtd">
<document>
  <header level="2">
    <text>Test</text>
  </header>
  <paragraph>
    <text>My list:</text>
  </paragraph>
  <list type="bullet" tight="true">
    <item>
      <paragraph>
        <text>foo</text>
      </paragraph>
    </item>
    <item>
      <paragraph>
        <text>bar</text>
      </paragraph>
    </item>
  </list>
</document>

Most of the value in commonmark and is probably in the latter. There already exist a few nice markdown converters for R including the popular rmarkdown package, which uses pandoc to convert markdown to several presentation formats.

The formal commonmark spec makes markdown suitable for more strict documentation purposes, where we might currently be inclined to use json or xml. For example we could use it to parse NEWS.md files from R packages in a way that allows for archiving and indexing individual news items, without ambiguity over indentation rules and such.

Every now and then a commercial software vendor makes claims on social media about how their software is so much better than the forecast package for R, but no details are provided.

There are lots of reasons why you might select a particular software solution, and R isn’t for everyone. But anyone claiming superiority should at least provide some evidence rather than make unsubstantiated claims.

The M3 forecasting competition was organized by Spyros Makridakis and Michèle Hibon. Entrants had to forecast 3003 time series and the results were compared to a test set that was withheld from participants. All the data (both training and test sets) and the forecasts of the original participants are publicly available in the Mcomp package for R. The competition results are also publicly available in an IJF paper published in 2000. The best performing methods overall were the Theta method, ForecastPro and ForecastX, as measured by the symmetric MAPE (sMAPE) that was favoured by Makridakis and Hibon. The following table shows some of the results from the original competition including results from the main commercial software vendors. The first sMAPE column is taken from the original paper. My own recalculation of the sMAPE results usually gives values slightly less than those published (I don’t know why). The MAPE column shows the mean absolute percentage error and the MASE column shows the mean absolute scaled errors.

BJ automatic was produced by ForecastPro but with the forecasts restricted to ARIMA models. For some reason, Autobox was allowed three separate submissions (a practice normally not allowed as it leads to over-fitting on the test set).

Any good forecasting software should be aiming to get close to (or better than) Theta on this test. After all, the M3 competition was held more than 15 years ago. Presumably all of the software companies have tried to improve their results since then. Unfortunately, none of them to my knowledge has published any updated figures. I wish they would (preferably independently verified). It would provide some evidence that they are improving their algorithms.

My aim with the forecast package for R is to make freely available state-of-the-art algorithms for some forecasting models. I do not attempt to offer a comprehensive suite of algorithms, but what I do provide gives forecasts that are in the same ballpark as the best methods in the M3 competition. Here is the evidence.

The last method is a simple average of the forecasts from ets and auto.arima. If you only want point forecasts, that is the best approach available in the forecast package. It is also better than any of the commercial software (at least as far as they have been prepared to subject their algorithms to independent testing).

Unlike the commercial vendors, you don’t have to take my word for it. My algorithms are open source, and the code that produced the above tables can be downloaded.

At the risk of coming across as even more of a curmudgeonly old fart than people already think I am, I really do dislike the current vogue in R that is the pipe family of binary operators; e.g. %>%. Introduced by Hadley Wickham and popularised and advanced via the magrittr package by Stefan Milton Bache, the basic idea brings the forward pipe of the F# language to R. At first, I was intrigued by the prospect and initial examples suggested this might be something I would find useful. But as time has progressed and I’ve seen the use of these pipes spread, I’ve grown to dislike the idea altogether. here I outline why.

The forward pipe operator is designed, in R at least (I’m not familiar with F#), to avoid the sort of nested/inline R code of the type shown below

the_data <- head(transform(subset(read.csv('/path/to/data/file.csv'),
                                  variable_a > x),
                           variable_c = variable_a/variable_b),
                 100)

replacing that awful mess with

the_data <-
  read.csv('/path/to/data/file.csv') %>%
  subset(variable_a > x) %>%
  transform(variable_c = variable_a/variable_b) %>%
  head(100)

And when compared against one another like that, who wouldn’t rejoice at the prospect of a pipe to banish such awful R code to distant memory? The problem with this comparison though is, who writes code like that in the first code block? I don’t think I’ve ever written code like that, even when I was a very green useR around the turn of the century.

When you compare the pipe version with how I’d lay out the R code

the_data <- read.csv('/path/to/data/file.csv')
the_data <- subset(the_data, variable_a > x)
the_data <- transform(the_data, variable_c = variable_a/variable_b)
the_data <- head(the_data, 100) # I'm perplexed as to why this would be a good thing to do?

the benefits of the pipe remain but they aren’t, at least in my opinion, as compelling. My version is verbose; I repeatedly overwrite the_data object with subsequent operations. Rather the writing the_data once in the pipe version, I’d write it 7 times! But that said, I could pass my version to a relative novice useR and they’d have a reasonable grasp of what the code did. I don’t think the same could be said for the pipe version.

But all that really doesn’t matter does it. It’s personal preference as to how you choose to write your data analysis and manipulation R script code. If you find it easier to write code and then read it back using the pipe operator all power to you.

Where I think it does make a difference is where you are

1. writing code to go into an R package for general consumption on say CRAN, or
2. writing example material for your package in a vignette or similar document.

I don’t claim that these are the only problem areas nor that these are universally accepted. I wager I’m in the majority position at the moment, but that is probably down to the relatively recent arrival of the pipe on the R scene.

Why is the pipe a problem if you are writing code to go into a general purpose R package that you expect users to abuse with their own data in their own code? Two reasons. The pipe operator involves the standard non-standard evaluation (NSE) paradigm. The pipe captures expressions on each side of the %>% operator and then arranges for the thing on the left of %>% to be injected into the expression on the right of %>%, usually as the first argument but not always. This all involves capturing the expressions and evaluating them within the %>%() function.

OK, isn’t that what all functions using a formula do, or what transform(), subset(), et al do? Well yes, and this is where my spider sense starts tingling. Who among us hasn’t had those things fail on us when we dropped them into an lapply() inside an anonymous function? Or wrapped those function as part of a package function only for some user to execute your function in a way you didn’t envisage? Now Hadley assures us that there is a correct way to do NSE and he even has a package for that, lazyeval. But still I have my reservations, despite Stefan’s attempts to allay my fears

@ucfagls @kevin_ushey @JennyBryan @noamross @Voovarb so far none have. You’re welcome to reopen the github issue if you have examples.

— Stefan Milton Bache (@stefanbache) May 28, 2015

OK, let’s assume Stefan and Hadley know what they are doing (and I invariably do) and the NSE used here really is safe. That still leaves the major problem I have with writing R code like this in package functions; how do you read it, parse it, and understand what it does? How do you track down a bug in the code and where it occurs if several steps are conflated into a single pipe chain? I’m not a pipe smoker so I’ll have to guess; you undo the chain and see where things break. Wouldn’t it have been easier to just write out the steps in the first place? That way the debugger can just step through the statements line by line as you’ve written them. I’m not alone in having concerns in this general area

@daattali @emhrt_ @ucfagls @noamross @recology_ @JennyBryan @Voovarb my main worry is that it makes errors harder to understand

— Hadley Wickham (@hadleywickham) May 28, 2015

I suppose a lot of this will come down to how well you grok pipes and how well you understand your actual code.

OK, enough of that; on to problem area number 2. I was recently helping a StackOverflow user massage some output from a vegan function into a format suitable for plotting with ggplot2. There, the aim was to go from this:

Group.1                S.obs     se.obs    S.chao1   se.chao1
Cliona celata complex  499.7143  59.32867  850.6860  65.16366
Cliona viridis         285.5000  51.68736  462.5465  45.57289
Dysidea fragilis       358.6667  61.03096  701.7499  73.82693
Phorbas fictitius      525.9167  24.66763  853.3261  57.73494

to this:

                Group.1   var        S       se
1 Cliona celata complex chao1 850.6860 65.16366
2 Cliona celata complex   obs 499.7143 59.32867
3        Cliona viridis chao1 462.5465 45.57289
4        Cliona viridis   obs 285.5000 51.68736
5      Dysidea fragilis chao1 701.7499 73.82693
6      Dysidea fragilis   obs 358.6667 61.03096
7     Phorbas fictitius chao1 853.3261 57.73494
8     Phorbas fictitius   obs 525.9167 24.66763

(or at least something pretty close it) so that the required dynamite plot (yes, yes, I know!) could be produced.

A little fiddling with reshape2 suggested this wasn’t something that it would handle gracefully (I may well be wrong here; I’m not familiar that particular package) and having recalled some details of Hadley’s tidyr package I felt that it would be more suited to the problem at hand. Not having used tidyr I proceeded to CRAN to grab the manual and look at any vignettes that might help me with understanding how to solve this particular problem. Thankfully, Hadley is a conscientious R package maintainer and there was a rather nice HTML-rendered version of the vignette right there on CRAN for me to peruse. The only downside to this was all the example code used pipes.

The very first usage example is (or was, depending on when you are reading this)

library(tidyr)
library(dplyr)
preg2 <- preg %>% 
  gather(treatment, n, treatmenta:treatmentb) %>%
  mutate(treatment = gsub("treatment", "", treatment)) %>%
  arrange(name, treatment)
preg2

Innocuous enough I guess, until you realise that I“m also reading the manual which has usage that doesn’t involve pipes and that Hadley isn’t naming the arguments in the calls here. Now I am having to grok what is being passed, and where, by the pipes, whilst trying to match the usage shown in the example snippet with the arguments in the manual. I might be old-school but yes, I do read the manual.

The point I’m trying to make here with my little anecdote is this; what point did the use of the pipe serve here? How am I as a user new to the package helped by Hadley also using the pipe? In my case I wasn’t; in fact it made it somewhat trickier to understand what went where, what the actual tidyr calls were etc. Now I fully understand that Hadley finds the pipe operator to be very expressive for data analysis, and who am I to argue with that? Where I would raise an issue is that if you are writing introductory example code, don’t force your users to have to grapple with two new concepts at once, at least not in the first few examples.

I don’t want to beat on Hadley over this; it’s just that this was a prime example of where the use of the pipe was obfuscatory not revelatory, for me at least.

So yes, I am a curmudgeonly old fart, but this old dog can learn new tricks. Convince me I’m wrong here cause I really do want to like the pipe; my Granddad smoked one and I have fond memories of the smell and, well, all the cool kids are using the pipe so it must be good, right?

Today is my lucky day.  I learned of two very interesting Web pages, both of them quite informative and the first of them rather provocative (yay!). I have some comments on both, in some cases consisting of mild disagreement, which I may post later, but in any event, I highly recommend both.  Here they are:

• Drew Schmidt’s take on parallel computation.
• Notes by Michael Kane and Bryan Lewis on GLM.

Tip: Read Drew’s page as soon as possible, before he removes the F-bombs. :-)

When fighting the good cyber-fight, one often has to process domain names. Our good friend @alexcpsec was in need of Punycode/IDNA processing in R which begat the newly-minted punycode R package. Much of the following has been culled from open documentation, so if you are already “in the know” about Punycode & IDNA, skip to the R code bits.

What is ‘Punycode’?

Punycode is a simple and efficient transfer encoding syntax designed for use with Internationalized Domain Names in Applications. It uniquely and reversibly transforms a Unicode string into an ASCII string. ASCII characters in the Unicode string are represented literally, and non-ASCII characters are represented by ASCII characters that are allowed in host name labels (letters, digits, and hyphens).

What is ‘IDNA’?

Until now, there has been no standard method for domain names to use characters outside the ASCII repertoire. The IDNA document defines internationalized domain names (IDNs) and a mechanism called IDNA for handling them in a standard fashion. IDNs use characters drawn from a large repertoire (Unicode), but IDNA allows the non-ASCII characters to be represented using only the ASCII characters already allowed in so-called host names today. This backward-compatible representation is required in existing protocols like DNS, so that IDNs can be introduced with no changes to the existing infrastructure. IDNA is only meant for processing domain names, not free text.

Why domain validation?

Organizations that manage some Top Level Domains (TLDs) have published tables with characters they accept within the domain. The reason may be to reduce complexity that come from using the full Unicode range, and to protect themselves from future (backwards incompatible) changes in the IDN or Unicode specifications. Libidn (and, hence, this package) implements an infrastructure for defining and checking strings against such tables.

Working with punycode

All three functions in the package are vectorized at the C-level.

For encoding and decoding operations, you pass in vectors of domain names and the functions return encoded or decoded character vectors. If there are any issues during the conversion of a particular domain name, the function will substitute Invalid for the domain name.

For the TLD validation function, any character set or conversion issue will cause FALSE to be returned. Otherwise the function will return TRUE.

Usage

devtools::install_github("hrbrmstr/punycode")
library(punycode)

ascii_doms <- c("xn------qpeiobbci9acacaca2c8a6ie7b9agmy.net",
"xn----0mcgcx6kho30j.com",
"xn----9hciecaaawbbp1b1cd.net",
"xn----9sbmbaig5bd2adgo.com",
"xn----ctbeewwhe7i.com",
"xn----ieuycya4cyb1b7jwa4fc8h4718bnq8c.com",
"xn----ny6a58fr8c8rtpsucir8k1bo62a.net",
"xn----peurf0asz4dzaln0qm161er8pd.biz",
"xn----twfb7ei8dwjzbf9dg.com",
"xn----ymcabp2br3mk93k.com")

intnl_doms <- c("ثبت-دومین.com",
"טיול-לפיליפינים.net",
"бизнес-тренер.com",
"новый-год.com",
"東京ライブ-バルーンスタンド.com",
"看護師高収入-求人.net",
"ユベラ-贅沢ポリフェノール.biz",
"เด็ก-ภูเก็ต.com",
"ایران-هاست.com")


for_valid <- c("gr€€n.no", "זגורי-אימפריה-לצפייה-ישירה.net", "ثبت-دومین.com",
"טיול-לפיליפינים.net", "xn------qpeiobbci9acacaca2c8a6ie7b9agmy.net", "xn----0mcgcx6kho30j.com",
"xn----9hciecaaawbbp1b1cd.net", "rudis.net")

# encoding

puny_encode(ascii_doms)

##  [1] "זגורי-אימפריה-לצפייה-ישירה.net"  "ثبت-دومین.com"                   "טיול-לפיליפינים.net"            
##  [4] "бизнес-тренер.com"               "новый-год.com"                   "東京ライブ-バルーンスタンド.com"
##  [7] "看護師高収入-求人.net"           "ユベラ-贅沢ポリフェノール.biz"   "เด็ก-ภูเก็ต.com"                      
## [10] "ایران-هاست.com"

# decoding

puny_decode(intnl_doms)

## [1] "xn----0mcgcx6kho30j.com"                   "xn----9hciecaaawbbp1b1cd.net"             
## [3] "xn----9sbmbaig5bd2adgo.com"                "xn----ctbeewwhe7i.com"                    
## [5] "xn----ieuycya4cyb1b7jwa4fc8h4718bnq8c.com" "xn----ny6a58fr8c8rtpsucir8k1bo62a.net"    
## [7] "xn----peurf0asz4dzaln0qm161er8pd.biz"      "xn----twfb7ei8dwjzbf9dg.com"              
## [9] "xn----ymcabp2br3mk93k.com"

# validation

puny_tld_check(for_valid)

## [1] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE

Fin

If you find any errors or need more functionality, post an issue on github and/or drop a note in the comments.

Today the course creation team at DataCamp released a new online R tutorial called Intermediate R. It is the sequel to our infamous Introduction to R course that has been taken by over 60,000 R enthusiasts. This new tutorial combines short videos with in-browser coding exercises to increase your R knowledge even more.

Start the new Intermediate R course for free, today.

What you’ll learn

The 6 hour Intermediate R tutorial is the logical next stop on your journey in the R programming language. In five chapters, each with short videos and challenging coding exercises inside your browser, you learn how to write better and more advanced R code for your daily data analysis tasks. Get a taste of conditionals and control flow in R, discover how and when to make use of the for and while loop, and get yourself familiar with writing your own functions in R.

After mastering the above topics, the tutorial continues with the apply family. Functions like lapply, sapply and vapply might seem hard to understand in the beginning, but once you master them they will become part of your daily routine for sure. This tutorial will take you step-by-step through the different members of the apply family and you will learn how to use them efficiently and effectively.

Having a solid knowledge of a wide range of R functions will definitely give you a competitive edge. That’s why the final part of the tutorial focuses less on programming concepts and more on what functions to use when facing common data science challenges. For example, learn how to make use of regular expressions in R to filter out certain observations, or discover how you can user R to analyze times and dates.

For those of you that are interested in earning a certificate we have some good news. Once you have finished the tutorial, you will earn a statement of accomplishment that you add directly to your linkedIn profile.

Start the course!

The post New online R tutorial by DataCamp: Intermediate R programming appeared first on The DataCamp Blog .