Change the normal hpfilter function to ignore na

Question

I am a new R user, trying to learn quickly, but I could not hack it myself. I work mainly with economic time series - therefore, try to save my dataset in multiple column format in xts format, for example:

> head(USDATAq) tq ngdp rgdp profit 1947 Q1 0 237.2 1770.7 20.7 1947 Q2 1 240.4 1768.0 23.9 1947 Q3 2 244.5 1766.5 23.8 1947 Q4 3 254.3 1793.3 25.5 1948 Q1 4 260.3 1821.8 29.4 1948 Q2 5 267.3 1855.3 31.2 

I use the hpfilter function to filter. Elsewhere on this site, I found this implementation that uses the coredata function to apply hpfilter objects to xts:

 hpfilter <- function(x, lambda=2){ eye <- diag(length(x)) dcrossprod <- crossprod(diff(eye, lag=1, d=2)) coredata(x) <- solve(eye + lambda * dcrossprod, coredata(x)) return(x) } 

My question is:

How can I change the function so that it works with variables that have NA observations (currently it calculates NA for the entire date range if there is any NA)?

I can pass the data set as na.omit(USDATAq) , which works, but it reduces all the variables in the data set to minimal observations. But different variables are available until different dates, and then NA. I would like to end up applying a function to each column of a dataset in a loop or mapply so that the function returns each filtered series using all available observations of that series.

Answer 1

I think you're on the right track. Why not just add na.omit inside this function? Just before creating the eye matrix? x<-na.omit(x) . Then all you need to do is transfer one-dimensional series, not whole data.frames. In other words: Leave the function as it is, add na.omit and combine it with lapply (or any form of the apply family (sapply, tapply, lapply) suits you best.

Answer 2

Thanks @ ran2. I worked on your proposal and managed to solve this problem, but in a rather difficult way. First of all, I could not get any "apply family" function to work correctly on the xts object, preserving its structure. The usual feed using (x, MARGIN = 2, ..) for column use showed promise, but settled on the coredata statement. lapply etc. gave distorted lists.

Then I went into the for loop. But since x <-na.omit (x) changes the length of the variable, it cannot replace the original loop.

 > for(i in 1:ncol(USDATAq)) { + USDATAq[,i]<-hpfilter(USDATAq[,i]) + } 

Error in NextMethod (.Generic): the number of elements to replace is not a multiple of the replacement length

So, I had to add obscene code to hpfilter to "merge the result back to the original (with NA) and then return the variable. This merge corresponds to two variables by date (hence length) filling NA into the result. Then this result can replace the original in the loop.In conclusion, I had to change hpfilter to:

 hpfilter <- function(x,lambda=2){ y<-na.omit(x) eye <- diag(length(y)) coredata(y) <- solve(eye + lambda * crossprod(diff(eye, lag=1, d=2)), coredata(y)) xy<-merge(x,y) return(xy[,2]) } 

and then use the loop above to finally get the results without errors. My knowledge of R is so rudimentary that there are probably simpler ways to do this. But at least I can continue. Thanks to everyone for pointing me in the right direction. I still welcome further corrections in my code above.

Answer 3

With zoo objects, it coredata() cleaned up a bit using attributes() rather than coredata() , after which you can hide directly into the zoo object. (I have not tried this for xts objects):

 hpfilter <- function(x,lambda=1600){ y<-na.omit(x) eye <- diag(length(y)) result <- solve(eye+lambda*crossprod(diff(eye,lag=1,d=2)),y) attributes(result) <- attributes(y) return(result) }