Non-linear modelling with several variables including a categorical variable

Hello everyone,

I am trying to model some data regarding a predator prey interaction experiment (n=26) in R. Predation rate is my response variable and I have 4 explanatory variables: predator density (1,2,3,4 5), predator size, prey density (5,10,15,20,25,30) and prey type (3 categories). I started with several linear models (glm) and found (as expected) that prey and predator density were non-linear related to predation rates. If I use a log transformation on these variables I get really nice curves and an adjusted R2 of 0.82, but it is not really the right approach for modelling non-linear relationships. Therefore I switched to non-linear least square regression (nls). I have several predator-prey models based on existing ecological literature e.g.:

model1 <- nls(rates ~ (a * prey)/(1 + b * prey), start = list(a = 0.27,b = 0.13), trace = TRUE) ### Holling's type II functional response

model2 <- nls(rates ~ (a*prey)/(1+ (b * prey) + c * (pred -1 )), start = list(a=0.22451, b=-0.18938, c=1.06941), trace=TRUE, subset=I1) ### Beddington-DeAngelis functional response

These models work perfectly, but now I want to add prey type as well. In the linear models prey type was the most important variable so I don't want to leave it out. I understand that you can't add categorical variables in nls, so I thought I try a generalized additive model (gam).

The problem with the gam models is that the smoothers (both spline and loess) don't work on both variables because there are only a very restricted number of values for prey density and predator density. I can manage to get a model with a single variable smoothed using loess. But for two variables it is simply not working. The spline function does not work at all because I have so few values (5) for my variables (see model 4).

model3 <- gam(rates~ lo(pred, span=0.9)+prey) ## this one is actually working but does not include a smoother for prey.

model4 <- gam(rates~ s(pred)+prey) ## this one gives problems:
A term has fewer unique covariate combinations than specified maximum degrees of freedom

My question is: are there any other possibilities to model data with 2 non-linear related variables in which I can also include a categorical variable. I would prefer to use nls (model2) with for example different intercepts for each category but I'm not sure how to get this sorted, if it is possible at all. The dataset is too small to split it up into the three categories, moreover, one of the categories only contains 5 data points.

Any help would be really appreciated.

With kind regards,
Robbie Weterings